Learning model replacement program, learning model replacement device, learning model replacement method, and neural network circuit device

Abstract

To provide a learning model replacement program, device, method and the like which improve the performance density in a calculator of a convolution neural network.SOLUTION: A method generates first teacher data DT from a plurality of two-dimensional input feature maps IM and each two-dimensional output feature map OM input to a first layer L of a first learning model according to the input of image data D0 for learning to the first learning model M, generates a second learning model R formed of a plurality of decision tree models by performing machine learning of the corresponding first teacher data for each of the plurality of two-dimensional output feature maps, specifies the specific two-dimensional output feature map OM in which determination accuracy of the second learning model satisfies a first condition in the map, generates a third learning model M1 by updating the first learning model such that the output of the specific two-dimensional output feature map in the first layer is performed by using the second learning model, and replaces the third learning model with the first learning model when the determination accuracy of the third learning model satisfies the second condition.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a learning model replacement program, a learning model replacement device, a learning model replacement method, and a neural network circuit device. [Background technology]

[0002] In recent years, there has been an increasing demand for implementing image recognition technology using convolutional neural networks (CNNs) on edge devices, including IoT (Internet of Things) devices. Therefore, various methods have been proposed in recent years, such as quantization of convolutional neural networks, with the aim of reducing the computational load in the inference process (e.g., reducing computational complexity and speeding up processing) (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Takumi Yatabe, Hiroshi Isobe, Yoichi Tomioka, and Hiroshi Saito, "A Study on Approximate Calculation of Binary Neural Networks Based on Decision Trees," Circuit and Systems Workshop, Work-in-Progress Session, pp. 121-123, August 2020. [Overview of the project] [Problems that the invention aims to solve]

[0004] In the quantization of convolutional neural networks described above, the quantization of feature maps and filter weights simplifies and miniaturizes the arithmetic units used in the sum-of-accumulate operations of the convolutional neural network (i.e., improves performance density).

[0005] However, in recent years, due to limitations in the manufacturing process of computing units, it has become difficult to achieve further improvements in performance density through quantization and other means.

[0006] Therefore, the object of the present invention is to provide a learning model replacement program, a learning model replacement device, a learning model replacement method, and a neural network circuit device that enable improvement of the performance density in the computing unit of a convolutional neural network. [Means for solving the problem]

[0007] To achieve the above objective, the learning model replacement program of the present invention performs the following processes on a computer: in response to the input of learning image data to a first learning model, it obtains a plurality of two-dimensional input feature maps input to the first layer constituting the first learning model and a plurality of two-dimensional output feature maps output from the first layer; for each of the plurality of two-dimensional output feature maps, it generates first training data from the plurality of two-dimensional input feature maps and each two-dimensional output feature map; for each of the plurality of two-dimensional output feature maps, it generates a second learning model consisting of a plurality of decision tree models by performing machine learning on the first training data corresponding to each two-dimensional output feature map; from the plurality of two-dimensional output feature maps, it identifies a specific two-dimensional output feature map whose judgment accuracy of the second learning model satisfies a first condition; it updates the first learning model to generate a third learning model so that the output of the specific two-dimensional output feature map in the first layer is performed using the second learning model; and if the judgment accuracy of the third learning model satisfies the second condition, it replaces the third learning model with the first learning model. [Effects of the Invention]

[0008] The learning model replacement program, learning model replacement device, learning model replacement method, and neural network circuit device of the present invention make it possible to improve the performance density of the arithmetic unit of a convolutional neural network. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 shows an example of the configuration of the information processing device 1 in the first embodiment. [Figure 2] Figure 2 is a diagram illustrating the schematic of the learning model replacement process in the first embodiment. [Figure 3] Figure 3 is a diagram illustrating the schematic of the learning model replacement process in the first embodiment. [Figure 4] Figure 4 is a diagram illustrating the schematic of the learning model replacement process in the first embodiment. [Figure 5] Figure 5 is a diagram illustrating the schematic of the learning model replacement process in the first embodiment. [Figure 6] Figure 6 is a flowchart illustrating the details of the learning model replacement process in the first embodiment. [Figure 7] Figure 7 is a flowchart illustrating the details of the learning model replacement process in the first embodiment. [Figure 8] Figure 8 is a flowchart illustrating the details of the learning model replacement process in the first embodiment. [Figure 9] Figure 9 is a flowchart illustrating the details of the learning model replacement process in the first embodiment. [Figure 10] Figure 10 is a flowchart illustrating the details of the learning model replacement process in the first embodiment. [Figure 11] Figure 11 is a diagram illustrating the details of the learning model replacement process in the first embodiment. [Figure 12] Figure 12 is a diagram illustrating the details of the learning model replacement process in the first embodiment. [Figure 13] Figure 13 shows an example of the configuration of the dedicated circuit 105 in the first embodiment. [Modes for carrying out the invention]

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, such example embodiments do not limit the technical scope of the present invention.

[0011] [Configuration example of information processing apparatus 1 in the first embodiment] First, a configuration example of an information processing apparatus 1 (hereinafter also referred to as a learning model replacement apparatus 1) in the first embodiment will be described. FIG. 1 is a diagram showing a configuration example of the information processing apparatus 1 in the first embodiment.

[0012] The information processing apparatus 1 is a computer apparatus, for example, a general-purpose PC (Personal Computer). And the information processing apparatus 1 performs a process of replacing at least a part of the layers of the elements constituting the learned learning model with those of another learning model (hereinafter also referred to as a learning model replacement process).

[0013] The information processing apparatus 1 has a hardware configuration of a general-purpose computer apparatus. For example, as shown in FIG. 1, it has a CPU 101 as a processor, a memory 102, a communication interface 103, a storage medium 104, and a dedicated circuit 105. Each part is connected to each other via a bus 106.

[0014] The storage medium 104 has, for example, a program storage area (not shown) for storing a program (not shown) for performing the learning model replacement process.

[0015] Also, the storage medium 104 has a storage area (not shown) for storing information used when performing the learning model replacement process. Note that the storage medium 104 may be, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0016] The CPU 101 performs the learning model replacement process by executing a program (not shown) loaded from the storage medium 104 into the memory 102.

[0017] The dedicated circuit 105 is, for example, an FPGA (Field Programmable Gate Array). The dedicated circuit 105 then performs, for example, inference processing using a learning model (hereinafter also simply referred to as inference processing), as will be described later.

[0018] The communication interface 103 communicates with the worker terminal 2 via a network NW, such as the Internet. The worker terminal 2 is, for example, a PC (Personal It may be a computer, and may be a terminal where an operator (hereinafter simply referred to as "operator") who generates a learning model inputs necessary information.

[0019] [Outline of the First Embodiment] Next, we will describe the outline of the first embodiment. Figures 2 to 5 are diagrams illustrating the outline of the first embodiment. Hereinafter, we will describe the outline of the first embodiment with reference to Figure 2.

[0020] As shown in Figure 2, the information processing device 1 implements various functions including an information management unit 111, a data generation unit 112, a first model generation unit 113, a map identification unit 114, a second model generation unit 115, a model replacement unit 116, a model output unit 117, and a model control unit 118. Specifically, the information management unit 111, the data generation unit 112, the first model generation unit 113, the map identification unit 114, the second model generation unit 115, the model replacement unit 116, and the model output unit 117 are functions that implement the learning model replacement process, while the model control unit 118 is a function that implements the inference process.

[0021] The information management unit 111 of the information processing device 1 stores various types of information entered by an operator via the operator terminal 2 in the storage medium 104. Specifically, when the information management unit 111 receives input of a learning model M (hereinafter also referred to as model M or first learning model M) as shown in Figure 3, it stores the received learning model M in the storage medium 104. Also, when the information management unit 111 receives input of image data D0 (hereinafter also referred to as learning image data D0) as shown in Figure 3, it stores the received image data D0 in the storage medium 104.

[0022] The data generation unit 112 of the information processing device 1 generates, for example, the training data DT shown in Figure 3 (hereinafter also referred to as the first training data DT).

[0023] Specifically, as shown in Figure 3, the data generation unit 112 acquires, for example, multiple 2D input feature maps IM input to a designated layer L (hereinafter also referred to as the first layer) that constitutes the learning model M, and multiple 2D output feature maps OM output from the designated layer L (multiple 2D output feature maps OM output by using each of the multiple filters in the designated layer L), in response to the input of image data D0 to the learning model M. The designated layer L is, for example, For example, it is a layer predetermined by the operator, and may consist of multiple consecutive layers. Hereafter, each of the multiple 2D output feature maps (OM) will also be referred to as a channel.

[0024] Then, the data generation unit 112 generates training data DT for each of the multiple two-dimensional output feature maps OM, for example, from the multiple two-dimensional input feature maps IM corresponding to each two-dimensional output feature map OM and each two-dimensional output feature map OM.

[0025] More specifically, as shown in Figure 3, the data generation unit 112 generates training data DT such that it includes, for example, pixels OPa (for example, data indicating "1" or "0") in multiple two-dimensional output feature maps OM and multiple pixels IPa in the two-dimensional input feature map IM used to calculate pixels OPa. The data generation unit 112 also generates training data DT such that it includes, for example, pixels OPb in multiple two-dimensional output feature maps OM and multiple pixels IPb in the two-dimensional input feature map IM used to calculate pixels OPb. Hereinafter, pixels in multiple two-dimensional output feature maps OM, including pixels OPa and pixels OPb, will be collectively referred to as pixels OP. Also, hereafter, multiple pixels IPa and multiple pixels IPb will be collectively referred to as multiple pixels IP or receptive field IP.

[0026] In other words, the data generation unit 112 generates training data DT such that, for each pixel OP in a plurality of two-dimensional output feature maps OM, each pixel OP and the plurality of pixels IP in the two-dimensional input feature map IM used to calculate each pixel OP are included.

[0027] The learning model M may, for example, receive multiple image data D0 sequentially as input. The data generation unit 112 may, for example, continuously generate multiple training data DT corresponding to each of the multiple image data D0.

[0028] The first model generation unit 113 generates a random forest model R (hereinafter also called the second learning model R) by, for example, learning (machine learning) the training data DT corresponding to each of the multiple 2D output feature maps OM output from a designated layer L.

[0029] Specifically, as shown in Figure 3, the first model generation unit 113 generates a random forest model R that can output (predict) one pixel OP contained in one two-dimensional output feature map OM from, for example, 16 pixels IP (4x4 pixel IPs) contained in each of three two-dimensional input feature maps IM. A specific example of the random forest model R will be explained below.

[0030] [Example of a Random Forest Model R Configuration] First, we will explain an example of the configuration of the decision tree model T that constitutes the random forest model R. Figure 4 shows an example of the configuration of the decision tree model T that constitutes the random forest model R. Hereafter, we will assume that the decision tree model T is a learning model that outputs one pixel OP from multiple pixels IP contained in each of the four 2D input feature maps IM. Furthermore, we will assume that the multiple pixels IP consist of 16 pixels (4x4 pixels).

[0031] The decision tree model T shown in Figure 4, for example, responds to inputs such as the pixel corresponding to X[2,0,0] (the third pixel from the left and the first pixel from the top among multiple pixel IPs included in the first 2D input feature map IM), the pixel corresponding to X[0,3,1] (the first pixel from the left and the fourth pixel from the top among multiple pixel IPs included in the second 2D input feature map IM), and the pixel corresponding to X[3,2,0] (the fourth pixel from the left and the third pixel from the top among multiple pixel IPs included in the first 2D input feature map IM) by determining the pixel OP. This is a learning model that outputs the inference results.

[0032] Specifically, the decision tree model T shown in Figure 4 predicts that pixel OP is "0" if, for example, it determines that the pixel corresponding to X[2,0,0] is not greater than "0" and the pixel corresponding to X[0,3,1] is not greater than "0". Also, the decision tree model T shown in Figure 4 predicts that pixel OP is "1" if, for example, it determines that the pixel corresponding to X[2,0,0] is not greater than "0" and the pixel corresponding to X[0,3,1] is greater than "0". Furthermore, the decision tree model T shown in Figure 4 predicts that pixel OP is "1" if, for example, it determines that the pixel corresponding to X[2,0,0] is greater than "0" and the pixel corresponding to X[3,2,0] is not greater than "0".

[0033] Therefore, if the pixel corresponding to X[2,0,0] is "0" and the pixel corresponding to X[0,3,1] is "1", the decision tree model T shown in Figure 4 will output "1" as the inference result for pixel OP, as shown in Figure 4.

[0034] Here, the depth D of the decision tree model T shown in Figure 4 is "3". Therefore, in the example shown in Figure 4, when inferring pixel OP using the decision tree model T, the number of operations is limited to only "2" comparison and branch operations.

[0035] Therefore, when performing pixel OP inference using the decision tree model T, the information processing device 1 can significantly reduce the number of operations compared to when performing pixel OP inference using multiply-accumulate operations. As a result, in this case, the information processing device 1 can miniaturize the arithmetic unit mounted on the dedicated circuit 105, thereby improving the area efficiency of the arithmetic unit.

[0036] Next, we will explain an example of the configuration of the Random Forest Model R. Figure 5 shows an example of the configuration of the Random Forest Model R.

[0037] The random forest model R shown in Figure 5 is a learning model composed of multiple decision tree models T, and performs inference by majority voting of the outputs from the multiple decision tree models T. Specifically, the random forest model R shown in Figure 5 performs inference by majority voting of the outputs from decision tree model Ta, decision tree model Tb, and decision tree model Tc, for example.

[0038] As a result, the information processing device 1 can improve the accuracy of its judgment when recognizing image data D0 compared to when using only the decision tree model T described in Figure 4. Specifically, the information processing device 1 can improve, for example, its judgment accuracy (generalization performance) for unknown image data D0 compared to when using only the decision tree model T described in Figure 4.

[0039] Returning to Figure 2, the map identification unit 114 identifies, for example, a 2D output feature map OM (hereinafter also referred to as a specific 2D output feature map OM) from among multiple 2D output feature maps OM output from a designated layer L, in which the determination accuracy of the random forest model R satisfies a condition (hereinafter also referred to as the first condition).

[0040] The second model generation unit 115 updates the learning model M so that, for example, the output of a specific two-dimensional output feature map OM in a designated layer L is performed using a random forest model R, thereby generating a learning model M1 (hereinafter also referred to as the third learning model M1).

[0041] The model replacement unit 116, for example, the learning model M1 generated by the second model generation unit 115 The judgment accuracy is calculated. Then, the model replacement unit 116 determines, for example, whether the calculated judgment accuracy satisfies the second condition. If it determines that the calculated judgment accuracy satisfies the second condition, the model replacement unit 116 replaces, for example, the learning model M1 with the learning model M.

[0042] The model output unit 117 outputs, for example, the learning model M that has been replaced by the model replacement unit 116. Specifically, the model output unit 117 outputs, for example, the learning model M that has been replaced by the model replacement unit 116 to the operator terminal 2.

[0043] When the model control unit 118 receives, for example, image data D0 (hereinafter also referred to as verification image data D0), it inputs the received image data D0 to the learning model M (for example, the learning model M output by the model output unit 117). The model control unit 118 then obtains, for example, the labels output from the learning model M in response to the input of image data D0 (for example, information indicating the inference result about the type of object shown in image data D0).

[0044] Subsequently, the model control unit 118 outputs, for example, the acquired label. Specifically, the model control unit 118 outputs, for example, the acquired label to the worker terminal 2.

[0045] In other words, as shown in Figure 3 and other figures, the information processing device 1 in this embodiment generates a learning model M1 by replacing a designated layer L included in the learning model M with a random forest model R generated from training data DT generated from a two-dimensional input feature map input to the designated layer L and a two-dimensional output feature map OM output from the designated layer L.

[0046] As a result, the information processing device 1 can generate a learning model M1 that approximates the learning model M, and reduces the number of calculations required to recognize image data D0 compared to the learning model M. Therefore, by using the learning model M1 instead of the learning model M, the information processing device 1 can further reduce the computational load required to recognize image data D0 while suppressing a decrease in judgment accuracy. Consequently, the information processing device 1 can miniaturize the arithmetic unit that performs calculations associated with the recognition of image data D0 while suppressing a decrease in judgment accuracy, and further improve the area efficiency of the arithmetic unit.

[0047] Furthermore, the information management unit 111, data generation unit 112, first model generation unit 113, map identification unit 114, second model generation unit 115, model replacement unit 116, and model output unit 117 may be implemented, for example, through the cooperation of the CPU 101 and a program. Also, the model control unit 118 may be implemented, for example, in a dedicated circuit 105.

[0048] Furthermore, the following description will focus on the case where a dedicated circuit 105 for performing inference processing is mounted on the information processing device 1. However, the dedicated circuit 105 may be mounted on an information processing device different from the information processing device 1. In other words, the inference processing in this embodiment may be performed on an information processing device different from the information processing device 1 that performs the model replacement processing in this embodiment.

[0049] [Details of the first embodiment] Next, we will describe the details of the learning model replacement process in the first embodiment. Figures 6 to 10 are flowcharts illustrating the details of the learning model replacement process in the first embodiment. Figures 11 and 12 are diagrams illustrating the details of the learning model replacement process in the first embodiment.

[0050] [Learning Model Memory Processing] First, we will explain the process of storing the learning model M in the storage medium 104 (hereinafter also referred to as the learning model storage process) as part of the learning model replacement process. Figure 6 is a diagram illustrating the learning model storage process.

[0051] As shown in Figure 6, the information management unit 111 waits, for example, until it receives input from the learning model M (NO in S1). Specifically, the information management unit 111 waits, for example, until the operator inputs the learning model M via the operator terminal 2.

[0052] Then, if input for the learning model M is received (YES in S1), the information management unit 111 stores the learning model M that was received in the storage medium 104 (S2).

[0053] [Image data storage processing] Next, we will explain the process of storing image data D0 in the storage medium 104 (hereinafter also referred to as the image data storage process) as part of the learning model replacement process. Figure 7 is a diagram illustrating the image data storage process.

[0054] As shown in Figure 7, the information management unit 111 waits, for example, until it receives input of image data D0 (NO in S11). Specifically, the information management unit 111 waits, for example, until the operator inputs image data D0 via the operator terminal 2.

[0055] Then, if the image data D0 is received (YES in S11), the information management unit 111 stores the received image data D0 in the storage medium 104 (S12).

[0056] In other words, the information management unit 111 stores the received image data D0 in the storage medium 104 each time it receives an input of image data D0, thereby accumulating multiple image data D0s in the storage medium 104. Also, if the information management unit 111 receives input of multiple image data D0s, it stores the multiple image data D0s that were received in the storage medium 104.

[0057] [Main process for replacing the learned model] Next, we will explain the main process of the learning model replacement process. Figures 8 to 10 are diagrams illustrating the main process of the learning model replacement process.

[0058] As shown in Figure 8, the information management unit 111 stores, for example, a duplicate of the learning model M stored in the storage medium 104, called learning model M0, in the storage medium 104 (S21).

[0059] Then, the information management unit 111 retrieves, for example, one of the designated layers L included in the designated layer list LS stored in the storage medium 104 that has not yet been retrieved in the process of S22 (S22). The designated layer list LS is, for example, a list that stores information indicating one or more designated layers L.

[0060] Specifically, the information management unit 111 retrieves the specified layers L included in the specified layer list LS, for example, so that the retrieval order proceeds from the first layer (shallow layer) to the last layer (deep layer) in the learning model M, and so that the retrieval order proceeds from the last layer (deep layer) to the first layer (shallow layer) in the learning model M. A concrete example of the specified layer list LS will be explained below.

[0061] [Specific example of a specified layer list LS] Figure 11 illustrates a specific example of the specified layer list LS.

[0062] The specified layer list LS shown in Figure 11 indicates, for example, that the first specified layer L consists of four layers including "Conv1", "BatchNorm1", "Quantization1", and "MaxPooling1", the second specified layer L consists of four layers including "Conv2", "BatchNorm2", "Quantization2", and "MaxPooling2", and the third specified layer L consists of four layers including "Conv3", "BatchNorm3", "Quantization3", and "MaxPooling3".

[0063] Therefore, in processing S22, the information management unit 111 acquires four layers, including "Conv1", "BatchNorm1", "Quantization1", and "MaxPooling1", as the designated layer L.

[0064] Returning to Figure 8, the data generation unit 112, for example, inputs multiple image data D0 stored in the storage medium 104 to the learning model M stored in the storage medium 104, and for each of the multiple image data D0, obtains multiple 2D input feature maps IM and multiple 2D output feature maps OM for the specified layer L obtained in the processing of S22 (S23).

[0065] Specifically, the data generation unit 112, for example, in response to the input of multiple image data D0 to the learning model M, obtains a combination for each of the multiple image data D0s: multiple 2D input feature maps IM that were input to the designated layer L obtained in the processing of S22, and multiple 2D output feature maps OM (multiple 2D output feature maps OM that were output by using each of the multiple filters in the designated layer L) that were output from the designated layer L obtained in the processing of S22.

[0066] Then, the data generation unit 112 sets the variable i to 1, for example (S24).

[0067] Next, the data generation unit 112 generates training data DT (hereinafter also referred to as training data DTi) for each of the multiple image data D0, for each pixel OP included in the two-dimensional output feature map OM corresponding to the i-th channel of the multiple two-dimensional output feature maps OM acquired in the processing of S23 (S25). This training data DT includes each pixel OP and multiple pixels IP of the two-dimensional input feature map IM used to calculate each pixel OP. In other words, the data generation unit 112 generates multiple training data DTi corresponding to each of the multiple image data D0.

[0068] Subsequently, the first model generation unit 113 generates a random forest model R (hereinafter also called the random forest model Ri) corresponding to the i-th channel by using, for example, a portion of the multiple training data DTi generated in the processing of S25 (hereinafter also referred to as the multiple training data Ti) (S26).

[0069] Furthermore, the map identification unit 114 calculates, for example, the judgment accuracy of the random forest model Ri generated in the processing of S26 (S27).

[0070] Specifically, the map identification unit 114 calculates the agreement rate Ei between the pixel OP output from the random forest model Ri in response to the input of other parts of the multiple training data DTi generated in the S25 process (hereinafter also referred to as multiple evaluation data Vi), and the pixel OP included in the multiple evaluation data Vi generated in the S25 process. The multiple evaluation data Vi may be, for example, multiple training data DTi generated in the S25 process that were not used as multiple training data Ti in the S26 process.

[0071] Next, the map identification unit 114, as shown in Figure 9, for example, the value set in variable i Add 1 to it (S31).

[0072] Then, the map identification unit 114 determines, for example, whether the value indicated by variable i has reached the number of channels corresponding to the specified layer L obtained in the S22 process (the number of multiple 2D output feature maps OM obtained in the S23 process) (S32).

[0073] As a result, if it is determined that the value indicated by variable i has not reached the number of channels corresponding to the specified layer L obtained in the S22 process (NO in S32), the information processing device 1 will, for example, repeat the processing from S25 onwards.

[0074] In other words, the information processing device 1 repeatedly performs the processes from S25 onward until it generates random forest models Ri corresponding to multiple channels corresponding to the specified layer L obtained in the S22 process, and then calculates the judgment accuracy corresponding to each of the generated random forest models Ri.

[0075] On the other hand, if the value indicated by variable i is determined to have reached the number of channels corresponding to the specified layer L obtained in the S22 process (YES in S32), the map identification unit 114 identifies, for example, the channel j corresponding to the random forest model Rj whose determination accuracy calculated in the S27 process satisfies the first condition, from among the channels corresponding to the specified layer L obtained in the S22 process (S33).

[0076] Specifically, the map identification unit 114 identifies, for example, the channel j corresponding to the random forest model Rj whose matching rate Ej calculated in the S27 process satisfies the first condition, from among the channels corresponding to the specified layer L obtained in the S22 process.

[0077] More specifically, the map identification unit 114 identifies, for example, the channel j corresponding to the random forest model Rj with the highest agreement rate Ej calculated in the S27, among the channels corresponding to the specified layer L obtained in the S22 process and not yet identified in the S33 process.

[0078] Then, the information management unit 111 stores, for example, a duplicate of the learning model M stored in the storage medium 104, called learning model M1, in the storage medium 104 (S34).

[0079] Furthermore, the second model generation unit 115 replaces, for example, the calculation performed by channel j, which is one of the multiple channels in the designated layer L that constitutes the learning model M replicated in the S34 process and was obtained in the S22 process, with an inference process performed by the random forest model Rj corresponding to channel j, which was identified in the S33 process (S35).

[0080] In other words, the second model generation unit 115 updates the learning model M so that, for example, the output of the two-dimensional output feature map OM by channel j identified in the S33 process from among the multiple channels in the designated layer L obtained in the S22 process is performed by the random forest model Rj corresponding to channel j identified in the S33 process.

[0081] Subsequently, the model replacement unit 116 calculates, for example, the judgment accuracy A1 of the learning model M that was replaced in the process of S35 (S36).

[0082] Specifically, the model replacement unit 116 takes, for example, the labels output from the learning model M1 in response to each input of multiple image data D0 (for example, information indicating the inference result about the type of object depicted in each of the multiple image data D0) and the ground truth labels corresponding to each of the multiple image data D0 (for example, the type of object depicted in each of the multiple image data D0). The accuracy A1 may be calculated as the rate of agreement with the information that shows the correct answer regarding the subject.

[0083] In other words, the model replacement unit 116 calculates the judgment accuracy A1 by using, for example, a portion of a plurality of training data (hereinafter also referred to as a plurality of second training data) which each contains image data D0 and the correct label corresponding to image data D0.

[0084] Subsequently, the model replacement unit 116, as shown in Figure 10, determines whether the judgment accuracy A1 calculated in the S36 process satisfies the second condition (S41).

[0085] Specifically, the model replacement unit 116 determines, for example, whether the judgment accuracy A1 calculated in the processing of S36 is equal to or greater than a predetermined threshold (hereinafter also referred to as the first threshold).

[0086] As a result, if the judgment accuracy A1 calculated in the S36 process determines that the second condition is not met (NO in S41), the information processing device 1 will, for example, repeat the processing from S33 onwards.

[0087] In other words, the map identification unit 114 repeatedly selects channels to replace with the random forest model Ri in order of the highest matching rate Ei calculated in the S27 process, until the judgment accuracy A1 calculated in the S36 process satisfies the second condition.

[0088] Specifically, as shown in Figure 12, the map identification unit 114 identifies, for example, the channel corresponding to the third 2D output feature map OM from the front and the channel corresponding to the fifth 2D output feature map OM from the front (the furthest 2D output feature map OM) as channels to be replaced with the random forest model Ri. The map identification unit 114 also identifies, for example, the channel corresponding to the first 2D output feature map OM from the front, the channel corresponding to the second 2D output feature map OM from the front, and the channel corresponding to the fourth 2D output feature map OM from the front as channels not to be replaced with the random forest model Ri.

[0089] As a result, the information processing device 1 in this embodiment can, for example, replace only the channels corresponding to the specified layer L acquired in the S22 process that are judged to have a small decrease in judgment accuracy when replaced with the random forest model Ri, with the random forest model Ri. Therefore, the information processing device 1 can, for example, suppress the decrease in judgment accuracy while reducing the number of calculations required to execute the inference process (recognition of image data D0).

[0090] On the other hand, if the judgment accuracy A1 calculated in the S36 process is determined to satisfy the second condition (YES in S41), that is, for example, if the judgment accuracy A1 calculated in the S36 process is determined to be less than the first threshold, the model replacement unit 116 performs fine tuning of the learning model M1, for example. Then, the model replacement unit 116 stores the finely tuned learning model M1 as the learning model M in the storage medium 104 (S42).

[0091] Specifically, the model replacement unit 116 may fine-tune the learning model M1 by, for example, retraining using another portion of the second training data (for example, the second training data not used in processing S36).

[0092] In addition, the model replacement unit 116 may, in the process of S41, determine whether the highest value among the matching rates Ei calculated in the process of S27 is equal to or greater than a predetermined threshold (hereinafter also referred to as the second threshold).

[0093] Then, if the highest value among the matching rates Ei calculated in the S27 process is determined to be equal to or greater than a predetermined threshold (NO in S41), the information processing device 1 may, for example, repeat the processing from S33 onwards.

[0094] On the other hand, if the highest value among the matching rates Ei calculated in the S27 process is determined not to be equal to or greater than a predetermined threshold (YES in S41), the information processing device 1 may, for example, perform the processing from S42 onwards.

[0095] Furthermore, the model replacement unit 116 may, in the process of S41, determine, for example, whether all channels corresponding to the specified layer L obtained in the process of S22 have been identified in the process of S33.

[0096] If the information processing device 1 determines that all channels corresponding to the specified layer L obtained in the S22 process have not been identified in the S33 process (NO in S41), it may, for example, repeat the processes from S33 onward.

[0097] On the other hand, if it is determined in the processing of S33 that all channels corresponding to the specified layer L obtained in the processing of S22 have been identified (YES in S41), the information processing device 1 may, for example, perform processing from S42 onwards.

[0098] Furthermore, in the S33 process, if it is determined that there is no channel j corresponding to the random forest model Rj whose determination accuracy calculated in the S27 process satisfies the first condition in the channel corresponding to the specified layer L obtained in the S22 process, the information processing device 1 may, for example, perform the S34 process and then proceed with the S42 and subsequent processes without performing the S35, S36, and S41 processes.

[0099] Subsequently, the model replacement unit 116 calculates, for example, the judgment accuracy A2 of the learning model M that was fine-tuned in the process of S42 (S43).

[0100] Specifically, the model replacement unit 116 may, similar to the processing in S36, calculate the degree of agreement between the label output from the learning model M for each input of multiple image data D0 and the correct label corresponding to each of the multiple image data D0, as the judgment accuracy A2.

[0101] Then, the model replacement unit 116 determines, for example, whether the judgment accuracy A2 calculated in the process of S43 satisfies the third condition (S44).

[0102] Specifically, the model replacement unit 116 determines, for example, whether the judgment accuracy A2 calculated in the S43 process is equal to or greater than a predetermined threshold (hereinafter also referred to as the third threshold).

[0103] As a result, if the judgment accuracy A2 calculated in the S36 process is determined to satisfy the third condition (YES in S44), the information management unit 111 stores, for example, a duplicate of the learning model M0 stored in the storage medium 104, in the storage medium 104 (S45).

[0104] On the other hand, if the judgment accuracy A2 calculated in the S43 process determines that the third condition is not met (NO in S44), the information processing device 1 does not perform the process in S45, for example.

[0105] Then, the information management unit 111 determines, for example, whether the designated layer list LS stored in the storage medium 104 is empty, that is, whether all designated layers L were obtained in the S22 process. The regulations were finalized (S46).

[0106] As a result, if it is determined that the designated layer list LS stored in the storage medium 104 is not empty (NO in S46), the information processing device 1 will, for example, repeat the processing from S21 onwards.

[0107] In other words, in this case, the information processing device 1 starts determining whether or not to replace the channels corresponding to other specified layers L included in the specified layer list LS with the random forest model R.

[0108] On the other hand, if the information processing device 1 determines that the specified layer list LS stored in the storage medium 104 is empty (YES in S46), it terminates the model replacement process, for example.

[0109] Specifically, in this case, the model output unit 117 outputs, for example, the learning model M0 stored in the storage medium 104 to the operator terminal 2.

[0110] In addition, the model replacement unit 116 may, for example, determine whether the designated layer list LS stored in the storage medium 104 is empty during the processing in S46.

[0111] If it is determined that a specified layer L included in the specified layer list LS exists (NO in S46), the information processing device 1 may, for example, repeat the processing from S21 onwards.

[0112] On the other hand, if it is determined that no specified layer L exists in the specified layer list LS (YES in S46), the information processing device 1 may, for example, terminate the model replacement process.

[0113] In other words, as shown in Figure 3 and other figures, the information processing device 1 in this embodiment generates a learning model M1 by replacing a designated layer L included in the learning model M with a random forest model R generated from training data DT generated from a two-dimensional input feature map input to the designated layer L and a two-dimensional output feature map OM output from the designated layer L.

[0114] As a result, the information processing device 1 can generate a learning model M1 that approximates the learning model M, and reduces the number of calculations required to recognize image data D0 compared to the learning model M. Therefore, by using the learning model M1 instead of the learning model M, the information processing device 1 can further reduce the computational load required to recognize image data D0 while suppressing a decrease in judgment accuracy. Consequently, the information processing device 1 can miniaturize the arithmetic unit that performs calculations associated with the recognition of image data D0 while suppressing a decrease in judgment accuracy, and further improve the performance density of the arithmetic unit.

[0115] [Example of the configuration of the dedicated circuit 105 in the first embodiment] Next, a specific example of the dedicated circuit 105 will be described. Figure 13 is a diagram showing an example of the configuration of the dedicated circuit 105 in the first embodiment.

[0116] The dedicated circuit 105 is a circuit that implements the random forest model Ri corresponding to the channel i generated in the processing of S23, that is, the random forest model Ri corresponding to the channel i replaced in the processing of S35, and includes, for example, a memory circuit SW and an arithmetic circuit RFA.

[0117] Furthermore, as shown in Figure 13 below, the number of multiple 2D input feature maps IM input to the random forest model Ri is C in The output from the random forest model Ri is as follows: The number of multiple 2D output feature maps OM is C out The explanation will proceed under the assumption that this is the case. Furthermore, the explanation below will proceed under the assumption that the number of pixels IP used to calculate one pixel OP included in multiple 2D output feature maps OM among multiple 2D input feature maps IM is R × R.

[0118] The memory circuit SW is, for example, a sliding window, and sequentially stores data corresponding to pixels included in a plurality of two-dimensional input feature maps IM input to the random forest model Ri. Specifically, the memory circuit SW stores, for example, data corresponding to pixels used in the operations in the random forest model Ri in order.

[0119] The arithmetic circuit RFA has, for example, C out (number) of random forest model circuits RTU. And each of the random forest model circuits RTU has, for example, T n (number) of decision tree model circuits DTU.

[0120] And when the arithmetic circuit RFA calculates pixels OP included in each of the C out (number) of two-dimensional output feature maps OM, among the data stored in the memory circuit SW, data (R×R×C in (number) of data) corresponding to the pixel OP is acquired, and the calculation of the pixel OP is performed in each of the C out (number) of random forest model circuits RTU. Specifically, in this case, each of the C out (number) of random forest model circuits RTU performs comparison operations and branch operations in each decision tree model circuit DTU constituting each random forest model circuit RTU to calculate the pixel OP corresponding to each random forest model circuit RTU.

[0121] That is, in the memory circuit SW, data corresponding to pixels included in a plurality of two-dimensional input feature maps IM is stored so that access to data required for calculating pixels OP included in each of the C out (number) of two-dimensional output feature maps OM can be performed simultaneously.

[0122] Thus, in the dedicated circuit 105, for example, a smaller comparator unit is installed instead of a multiplier, adder, and multiply-accumulate unit (hereinafter also referred to as "multiplier, etc."), making it possible to realize a circuit with a smaller area than when a multiplier, etc. is installed. Therefore, in the dedicated circuit 105, for example, it is possible to improve performance density.

[0123] Furthermore, if there are layers or channels that were not replaced with the random forest model Ri during the learning model replacement process, the dedicated circuit 105 may be equipped with multipliers or the like that perform sum-of-accumulate operations in those layers or channels. [Explanation of Symbols]

[0124] 1: Information Processing Device 2: Worker terminal 101:CPU 102: Memory 103: Communication Interface 104:Storage medium 105: Dedicated circuit 106: Bus 111: Information Management Department 112: Data Generation Unit 113: First Model Generation Unit 114: Map Identification Section 115: Second Model Generation Unit 116: Model replacement section 117: Model Output Section 118: Model Control Unit

Claims

1. In response to the input of training image data to the first learning model, a plurality of two-dimensional input feature maps input to the first layer constituting the first learning model and a plurality of two-dimensional output feature maps output from the first layer are acquired. For each of the aforementioned two-dimensional output feature maps, first training data is generated from the aforementioned two-dimensional input feature maps and each two-dimensional output feature map. For each of the aforementioned two-dimensional output feature maps, a second learning model consisting of multiple decision tree models is generated by performing machine learning on the first training data corresponding to each two-dimensional output feature map. Among the plurality of two-dimensional output feature maps, a specific two-dimensional output feature map is identified in which the determination accuracy of the second learning model satisfies the first condition. The first learning model is updated to generate a third learning model so that the output of the specific two-dimensional output feature map in the first layer is performed using the second learning model. If the judgment accuracy of the third learning model satisfies the second condition, the third learning model is replaced with the first learning model. A learning model replacement program characterized by having a computer perform the processing.

2. In claim 1, In the aforementioned process to be identified, The determination accuracy of the second learning model corresponding to each of the plurality of two-dimensional output feature maps is calculated. Among the plurality of two-dimensional output feature maps, the two-dimensional output feature map corresponding to the second learning model with the highest determination accuracy is identified as the specific two-dimensional output feature map. A learning model replacement program characterized by the following:

3. In claim 2, In the replacement process, if the determination accuracy corresponding to the specific two-dimensional output feature map is equal to or greater than a predetermined threshold, the third learning model is replaced with the first learning model. A learning model replacement program characterized by the following:

4. In claim 1, In the process of generating the second learning model, the second learning model is generated by performing machine learning on a portion of the first training data corresponding to each of the plurality of two-dimensional output feature maps. In the aforementioned identification process, the agreement rate between the multiple two-dimensional output feature maps output from the second learning model in response to the input of the multiple two-dimensional input feature maps included in the other part of the first training data and the multiple two-dimensional output feature maps included in the other part of the first training data is calculated as the judgment accuracy of the second learning model. A learning model replacement program characterized by the following:

5. In claim 1, further, Among the plurality of two-dimensional output feature maps other than the aforementioned specific two-dimensional output feature map, the second learning model identifies a new specific two-dimensional output feature map that satisfies the first condition. A new third learning model is generated by updating the third learning model so that the output of the new specific two-dimensional output feature map in the first layer is performed using the second learning model. Let the computer perform the process, In the replacement process described above, if the judgment accuracy of the new third learning model satisfies the second condition, the new third learning model is replaced with the first learning model. A learning model replacement program characterized by the following:

6. In claim 1, In the replacement process described above, the agreement rate between the label output from the third learning model in response to the input of the training image data included in the second training data, which includes the training image data and the correct label corresponding to the training image data, and the correct label included in the second training data, is calculated as the judgment accuracy of the third learning model. A learning model replacement program characterized by the following:

7. In claim 1, In the aforementioned replacement process, The agreement rate between the label output from the third learning model in response to inputting the training image data, which is included in a portion of the second training data that includes the training image data and the correct label corresponding to the training image data, and the correct label included in the portion of the second training data, is calculated as the judgment accuracy of the third learning model. If the judgment accuracy of the third learning model satisfies the second condition, the third learning model is retrained by performing machine learning on the remaining portion of the second training data. If the judgment accuracy of the retrained third learning model satisfies the third condition, the third learning model is replaced with the first learning model. A learning model replacement program characterized by the following:

8. A data generation unit that, in response to the input of training image data to a first learning model, acquires a plurality of two-dimensional input feature maps input to the first layer constituting the first learning model and a plurality of two-dimensional output feature maps output from the first layer, and generates first training data from the plurality of two-dimensional input feature maps and each two-dimensional output feature map for each of the plurality of two-dimensional output feature maps, A first model generation unit generates a second learning model consisting of multiple decision tree models by performing machine learning on the first training data corresponding to each of the multiple two-dimensional output feature maps, A map identification unit identifies a specific two-dimensional output feature map among the plurality of two-dimensional output feature maps such that the determination accuracy of the second learning model satisfies the first condition, A second model generation unit updates the first learning model to generate a third learning model so that the output of the specific two-dimensional output feature map in the first layer is performed using the second learning model, The system includes a model replacement unit that replaces the third learning model with the first learning model if the determination accuracy of the third learning model satisfies the second condition, A learning model replacement device characterized by the following:

9. In response to the input of training image data to the first learning model, a plurality of two-dimensional input feature maps input to the first layer constituting the first learning model and a plurality of two-dimensional output feature maps output from the first layer are acquired. For each of the aforementioned two-dimensional output feature maps, first training data is generated from the aforementioned two-dimensional input feature maps and each two-dimensional output feature map. For each of the aforementioned two-dimensional output feature maps, a second learning model consisting of multiple decision tree models is generated by performing machine learning on the first training data corresponding to each two-dimensional output feature map. Among the plurality of two-dimensional output feature maps, a specific two-dimensional output feature map is identified in which the determination accuracy of the second learning model satisfies the first condition. The first learning model is updated to generate a third learning model so that the output of the specific two-dimensional output feature map in the first layer is performed using the second learning model. If the judgment accuracy of the third learning model satisfies the second condition, the third learning model is replaced with the first learning model. A method for replacing a learning model, characterized in that the processing is performed by a computer.

Images