Data generation method, semiconductor integrated circuit, program, device and system

Abstract

To provide an information processing apparatus which executes machine learning efficiently.SOLUTION: An information processing apparatus according to one embodiment includes: obtaining means which obtains a source code including a code which defines Forward processing of each layer constituting a neural network; storage means which stores associations between each Forward processing and Backward processing corresponding to each Forward processing; and executing means which successively executes each code included in the source code, and which calculates an output value of the Forward processing defined by the codes based on an input value at the time of execution of each code, and generates a reference structure for Backward processing in a layer corresponding to the code based on the associations stored in the storage means.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The technology disclosed in this specification relates to machine learning.

Background Art

[0002] In recent years, machine learning using neural networks has been utilized in various fields. When performing such machine learning, developers, etc. create source code that defines the network structure, etc. of a neural network using a predetermined programming language, and by causing the personal computer, etc. to execute the source code thus created, it is possible to execute machine learning on such a personal computer (Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, there has been a need for a framework that enables efficient creation of source code that defines the network structure, etc. of a neural network.

Means for Solving the Problems

[0005] Therefore, according to various embodiments of the present invention, there are provided an information processing apparatus and an information processing method for efficiently executing machine learning.

[0006] An information processing apparatus according to one aspect includes: an acquisition unit that acquires source code including code defining forward processing of each layer constituting a neural network; a storage unit that stores a correspondence relationship between each forward processing and backward processing corresponding to the forward processing; and an execution unit that sequentially executes each code included in the source code. When each code is executed, the execution unit calculates an output value of the forward processing defined by the code based on an input value, and generates a reference structure for backward processing in a layer corresponding to the code based on the correspondence relationship stored in the storage unit. Also, a computer program according to one aspect causes a computer to function as: an acquisition unit that acquires source code including code defining forward processing of each layer constituting a neural network; a storage unit that stores a correspondence relationship between each forward processing and backward processing corresponding to the forward processing; and an execution unit that sequentially executes each code included in the source code. When each code is executed, the execution unit calculates an output value of the forward processing defined by the code based on an input value, and generates a reference structure for backward processing in a layer corresponding to the code based on the correspondence relationship stored in the storage unit.

Advantages of the Invention

[0007] According to various embodiments of the present invention, it is possible to provide an information processing apparatus and an information processing method for efficiently executing machine learning.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

[0009] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. Note that the same reference numerals are assigned to common components in the respective drawings. First, in the first part, an information processing apparatus according to an embodiment (hereinafter, described as a learning apparatus which is an example of the information processing apparatus) will be described, and in the second part, a method of implementing an algorithm implemented in the information processing apparatus according to the embodiment in an embedded chip (embedded semiconductor integrated circuit) will be described.

[0010] First part (Learning apparatus according to the embodiment) 1. Background and overview Machine learning algorithms including deep learning can often be formulated as a minimization problem of the sum of loss functions defined for each model. The loss function is an index represented by the error between the output of the model and the correct answer in a given learning data sample. Here, a series of processes from inputting data into the model to obtaining the output and comparing it with the correct answer is called a computational graph, and the result is used as the loss function. The minimization problem of the loss function can be solved by a general method called the gradient method as long as the gradient obtained by differentiating the loss function can be calculated.

[0011] When attempting to implement as a computer program, there is a method of coding the entire loss function and gradient by oneself. However, calculating the gradient of a complex model is generally difficult, and it is often difficult to obtain an explicit calculation formula and it cannot be directly described as a program. Therefore, the second method is to use a calculation library such as Caffe (http: / / caffe.berkeleyvision.org / ), Torch (http: / / torch.ch / ), Theano (http: / / deeplearning.net / software / theano / ). Note that the entire content disclosed at these URLs is incorporated herein by reference.

[0012] In these libraries, in a dedicated mini-programming language, the loss function can be described only as a combination of prepared basic computational elements (Primitives), and its gradient function can also be automatically obtained. This is because the gradient of each basic computational element itself is defined, so the gradient of the entire combination can also be obtained as automatic differentiation. That is, even a neural network that can be expressed as a large-scale computational graph as used in deep learning can be trained by the gradient method using its gradient function if the calculation of its loss function can be explicitly expressed using this mini-programming language.

[0013] Such a computational library has heretofore been based on a calculation procedure that the applicant refers to as "Define-and-Run". This is an approach where first, a computational graph is defined (Define), and after the gradient is derived by automatic differentiation, learning (Run) with learning data is advanced. This approach has advantages such as that a series of gradient calculations can be compiled and prepared in advance as a single unit for speedup at the time of Define when the computational graph does not have complex control statements (such as if and for) and does not change over time, and memory management is unnecessary.

[0014] However, in the case of a computational graph with complex control statements that has increased with the development of deep learning research, or in the case of a model where the computational graph dynamically changes even under meta-conditions that do not depend on data, there have been problems such as the low expressiveness of the mini-programming language, the difficulty of debugging, and the deterioration of memory efficiency due to the inability to dynamically change the structure. Therefore, depending on the complexity of the model and the scale of the data, implementation and execution may be difficult.

[0015] Therefore, in the embodiment, the applicant of the present application proposes a new calculation procedure called "Define-by-Run". Specifically, in the embodiment, instead of having a fixed graph structure in advance like "Define-and-Run", an approach is adopted in which the graph structure is dynamically extracted and memorized in each learning (Run), meta changes are made, and the gradients are recalculated each time.

[0016] As a result, there is an effect that a mini programming language for defining a graph in advance becomes unnecessary, and for developers, the costs of its design, implementation, and maintenance, and for users, the learning costs and debugging difficulties are removed. Also, regarding control statements, since those of general programming languages (C, Java (registered trademark), Python) can be freely used, neural networks with more complex graph structures can be easily implemented. Furthermore, by enabling certain conditional meta change operations on the graph, improvement in memory efficiency and flexible learning and application of the model are realized.

[0017] The conceptual difference between the method called "Define-and-Run" according to the prior art described above and the method called "Define-by-Run" according to the embodiment is also clear by comparing FIG. 1 and FIG. 2. FIG. 1 is a schematic diagram conceptually showing the method called "Define-and-Run" according to the prior art, and FIG. 2 is a schematic diagram conceptually showing the method called "Define-by-Run" according to the embodiment of the present invention. In the Define-and-run configuration shown in FIG. 1, first, a mini programming language inputs only the model definition and outputs the calculation procedures of the forward (identification) process and the backward (learning) process, which are the entities of the calculation graph (Define step). In the next step, the Foward / Backward processing system performs data input and parameter (weight) update according to the calculation procedures of the forward (identification) process and the backward (learning) process (Run step). In contrast, in the Define-by-run configuration shown in Figure 2, the processing system of a general-purpose programming language executes forward (identification) processing while inputting the model definition, input data, and parameters, and at the same time generates the calculation procedure for backward (learning) processing. Here, the model definition is defined in accordance with the grammar of a general-purpose programming language such as function calls, arithmetic operations, loops, and branches. The calculation procedure for backward (learning) processing can be dynamically changed independently of the execution of forward (identification) processing. The Backward processing system can be called at any timing. The Backward processing system updates the parameters from the input data and the result of the Foward processing according to the calculation procedure of Backward.

[0018] 2. Background Art Related to Neural Networks 2-1. Basic Processing Flow of Neural Networks The processing performed in a neural network mainly includes forward (Forward) processing, backward (Backward) processing, and weight update. Forward processing refers to the processing of processing and propagating information from the input layer to the output layer of a neural network.

[0019] Backward processing refers to performing two processes, error backpropagation and weight gradient calculation, from the output layer to the input layer of a neural network. Error backpropagation refers to the process of propagating the error (δ) obtained from the output-side layer to the input-side layer. Weight gradient calculation refers to the process of obtaining the weight gradient (∂W) from the error (δ) obtained from the output-side layer and the output value of the input-side layer for the layer having weights.

[0020] Weight update refers to the process of updating the weights for the layer having weights by an algorithm derived from stochastic gradient descent (SGD) using the weight gradient (∂W) obtained by the above weight gradient calculation. This weight update is executed once for each unit of batch processing.

[0021] 2-2. Frequently Appearing Calculation Modules in Examples of Neural Networks Each layer constituting the neural network is realized by, for example, the layer algorithms listed below. -Linear -ReLu -Dropout -Softmax Cross Entropy -Convolution 2D -Pooling (such as Average Pooling and Max Pooling, etc.)

[0022] Typical examples of weight update algorithms include the following. -momentum-SGD -Adam, etc.

[0023] 2-3. Network Configuration Example of Neural Network (1) Figure 3 is a schematic diagram showing an example of the network configuration of a neural network. In Figure 3, as an example, a neural network with six intermediate layers (Linear, ReLU, Linear, ReLU, Dropout, and Linear) arranged between the input layer and the output layer (Softmax) is illustrated. On the paper, the rightward arrow indicates the forward process, and the leftward arrow indicates the backward process. Since the input layer does not have weights to be updated, the backward process is performed up to the intermediate layer having the weights closest to this input layer (in the example shown in Figure 3, the Linear layer arranged adjacent to the input layer).

[0024] 2-4. Network Configuration Example of Neural Network (2) Figure 4 is a schematic diagram showing another example of the network configuration of a neural network. In Figure 4, as an example, a neural network in which a plurality of intermediate layers (Convolution 2D, ReLU, Convolution 2D, ReLU) arranged in series are arranged in parallel (three in number) is illustrated between an input layer and an intermediate layer (Linear) arranged adjacent to an output layer (Softmax). On the paper surface, an upward arrow indicates forward processing, and a downward arrow indicates backward processing.

[0025] 2-5. Example of Network Configuration of Neural Network (3) Figure 5 is a schematic diagram showing yet another example of the network configuration of a neural network. In Figure 5, as an example, a neural network having a loop (which may be called a "Recurrent Neural Network") is illustrated. In the figure, the flow of data in forward processing is indicated by arrows. The intermediate layer (here Linear) executes a calculation in which the sum of the output value of this intermediate layer in the previous time and the output value of the input layer in this time is used as the input of this intermediate layer. As a method for realizing backward processing in such a neural network, a method (BPTT) of expanding the network in the time axis direction in advance to convert it into a network without a loop is known.

[0026] 2-6. Calculation Contents of Layer Algorithm (Linear) One of the layer algorithms, Linear, executes a calculation that repeats the operation of taking the weighted average of all nodes in the input-side layer for the number of nodes in the intermediate layer. Figure 6 is a diagram showing pseudo-code for realizing the calculation executed during forward processing by Linear, and Figure 7 is a diagram showing pseudo-code for realizing the calculation executed during backward processing by Linear.

[0027] 2-7. Calculation Contents of Layer Algorithm (ReLU) ReLU, one of the layer algorithms, calculates Max(0,val) for each node in the input layer. This algorithm is the most widely used method in recent years for adding nonlinearity (activation function) to neural network calculations. FIG. 8 is a diagram showing pseudocode for implementing calculations performed by ReLU during forward processing, and FIG. 9 is a diagram showing pseudocode for implementing calculations performed by ReLU during backward processing.

[0028] 2-8.Layer Algorithm Calculation Contents (Dropout) Dropout, a layer algorithm, randomly selects a certain percentage of nodes and performs calculations to disable the output and backpropagation. This algorithm is unnecessary when only discrimination is performed (i.e., training is not performed). 2-9. Layer Algorithm Calculation Contents (Softmax Cross Entropy) Softmax Cross Entropy, one of the layer algorithms, corrects the values of the input layer using the following formula:

number

[0029] 2-10.Layer Algorithm Calculation Contents (Convolution 2D) One of the layer algorithms, Convolution 2D, convolves an image with a data structure of Channel*Width*Height. Both the input-side layer and the output of this layer have a data structure of Channel*Width*Height. In this algorithm, it is also possible to reduce the image size by stride processing. Also, in this algorithm, padding is inserted into the image of the input-side layer. This algorithm has a calculation structure similar to Linear in the Channel direction (repeating the inner product calculation of the input channels the number of output channels times). Figure 10 is a diagram showing pseudo-code for realizing the calculations executed during forward processing by Convolution 2D. Note that during backward processing, Convolution 2D executes weight gradient calculation and error backpropagation in the same way as Linear. The scale of the loop for each process is the same as that during forward processing.

[0030] 2-11. Calculation Contents of Layer Algorithm (Max Pooling) One of the layer algorithms, Max Pooling, reduces the image in the vertical and horizontal directions by taking the maximum value of the image of the input-side layer. Note that there may be cases where the filter size for taking the maximum value and the stride width for image reduction are different. Also, there is no change in the number of channels.

[0031] 2-12. Calculation Contents of Layer Algorithm (Average Pooling) One of the layer algorithms, Max Pooling, reduces the image in the vertical and horizontal directions by taking the average value of the image of the input-side layer. Note that there may be cases where the filter size for taking the average value and the stride width for image reduction are different. Also, there is no change in the number of channels.

[0032] 2-13. Weight Update Algorithm As weight update algorithms, there are various algorithms derived from Stochastic Gradient Descent (SGD). In these algorithms, the calculations are independent for each element of the weights. The calculation formula of momentum-SGD mentioned above is as follows.

Number

Number

[0033] 3. Hardware Configuration of the Learning Device According to the Embodiment Next, the hardware configuration of the learning device according to the embodiment of the present invention will be described. FIG. 11 is a schematic diagram showing a hardware configuration example of the learning device according to an embodiment of the present invention.

[0034] As shown in FIG. 11, the learning device 10 includes a CPU 11, a main memory 12, an input I / F 13, an output I / F 14, a communication I / F 15, an external memory 16, and a user I / F 17, and these components are electrically connected to each other via an internal bus 18. Note that the learning device 10 may optionally include a GPU (not shown).

[0035] The CPU 11 loads various programs such as an operating system and a program that supports a programming language (e.g., Python) (a program used for creating source code) from the external memory 16 into the main memory 12 and executes the instructions included in the loaded programs. The main memory 12 is used to store the programs executed by the CPU 11 and is composed of, for example, DRAM.

[0036] The input I / F 13 has a function of capturing the output data of a measuring device (not shown) and is connected to each component by the internal bus 18. Here, various measurement data that are the output of the measuring device include information acquired by sensors and the like, for example, temperature, humidity, position information, image data, etc., and may also be time-series data such as video data or a temperature data series acquired at certain intervals of temperature. The output I / F 14 receives data from each component through the internal bus 18 and outputs it to an output device (not shown) outside the learning device. Here, the data output to the output device is assumed to be, for example, control information when driving a motor, or control information for information output devices such as buzzers, control switches, the accelerator and brake of an automobile, and liquid crystal displays.

[0037] The communication I / F 15 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate various information with a server device (not shown) via the communication network 20. The external memory 16 is composed of, for example, a magnetic disk drive or a flash memory, and stores various programs such as an operating system and a program (a program used for creating source code) that supports a programming language (for example, Python).

[0038] The learning device 10 according to one embodiment having the above configuration can function as a learning device that performs machine learning when the CPU 11 (optionally, in addition to this, a GPU) executes a predetermined program loaded from the external memory 16 into the main memory 12. For example, the learning device 10 that performs machine learning can be realized as a learning device modeled by a neural network when the CPU 11 (optionally, in addition to this, a GPU) executes various programs.

[0039] The learning device 10 having the above configuration can be mounted on a corresponding individual (device). Further, the learning device 10 can be connected to a corresponding measuring device and a corresponding output device. These measuring device and output device may be mounted on a corresponding individual (device), or may be connected as separate devices using communication means.

[0040] In one embodiment, the learning device 10 is any information processing device capable of executing machine learning, and includes, for example, but is not limited to, a personal computer, a tablet, a mobile phone, a smartphone, a portable information terminal, a touch pad, and an information processing server, etc.

[0041] 4. Functional Blocks of the Learning Device According to the Embodiment Next, the functions of the learning device 10 having the above configuration will be briefly described. FIG. 12 is a block diagram schematically showing an example of the functions of the learning device according to an embodiment of the present invention.

[0042] The learning device 10 according to the embodiment is based on a method called "Define-by-Run" as described above. Specifically, the learning device 10 according to the embodiment has a mechanism that can actually execute the backward processing and weight update processing by dynamically generating the information on the network configuration necessary for the backward processing and weight update processing at the timing when the forward processing of the neural network is executed by a general procedural language including branches, loops, and function calls.

[0043] In order to implement such "Define-by-Run", as shown in FIG. 12, a learning device 10 according to an embodiment mainly includes an acquisition unit 110, a storage unit 120, and an execution unit 130. The acquisition unit 110 acquires source code including code that defines the forward processing of each layer constituting the neural network. Specifically, such source code is created by a developer, a user, or the like using a text editor in a predetermined programming language (such as Python), and the acquisition unit 110 acquires such source code. Such an acquisition unit 110 can be realized, for example, by the cooperation of the CPU 11, the main memory 12, the external memory 16, the user I / F 17, etc. shown in FIG. 11.

[0044] The storage unit 120 stores the correspondence between each of a plurality of definable forward processes in the source code and the backward process corresponding to the forward process. In the correspondence stored in the storage unit 120, for a certain forward process included in the plurality of forward processes, the corresponding backward process is associated in a one-to-one relationship. That is, in the correspondence stored in the storage unit 120, for example, for a layer (intermediate layer) called Linear, the forward process corresponding to Linear and the backward process corresponding to this forward process are associated. (Such a one-to-one correspondence between the forward process and the backward process is used to execute the process corresponding to the forward process when executing the backward process using the reference structure for the backward process. For example, when the forward process is executed in the order of A→B→C, the backward process is executed in the order of C→B→A. Here, since both the forward process and the backward process are implemented in pairs for each of the functions from A to C, such a backward process can be realized.) Note that the storage unit 120 can store various information including the source code acquired by the acquisition unit 110 and various libraries and the like used in the programming language corresponding to this source code. Such a storage unit 120 can be realized, for example, by the cooperation of the CPU 11, the main memory 12, the external memory 16, etc. shown in FIG. 11.

[0045] The execution unit 130 sequentially executes each code included in the source code acquired by the acquisition unit 110 (stored in the storage unit 120). When this execution unit 130 executes each code, it can calculate the output value of the forward process defined by the code based on the input value. Further, when this execution unit 130 executes each code, it can generate the reference structure between objects in the layer corresponding to the code. Such an execution unit 130 can be realized, for example, by the cooperation of the CPU 11, the main memory 12, the external memory 16, etc. shown in FIG. 11.

[0046] Also, in order to implement the above-described "Define-by-Run" method, the learning device 10 according to an embodiment uses the above-described acquisition unit 110, storage unit 120, and execution unit 130 to utilize three classes, namely, the three classes of Function, Variable, and Optimizer. Note that the names of these classes are given for convenience and are not restrictive. First, the class of Function is a class defined by pairing forward processing and backward processing. This class of Function defines the specific layer algorithms exemplified in the above "2-6" to "2-12" as subclasses. Next, the class Variable is a class that manages the data input and output between functions. This class Variable has the role of hiding the differences between GPUs and CPUs, and also has a method (unchain_backward, described later) for terminating the backward processing of a network including loops within a limited range. Furthermore, the class Optimizer is a class that updates weights.

[0047] 5. Operation Example 1 Next, a specific example of the operation performed by the learning device 10 according to the embodiment having the above configuration will be described. FIG. 13 is a diagram showing an example of source code input to the learning device according to an embodiment of the present invention. Note that it should be noted that the source code illustrated in FIG. 13 has been intentionally simplified for the purpose of explaining the features of the learning device according to the present embodiment. Also, the line numbers described at the left end in FIG. 13 are attached for explaining this specific example and are not included in the actual source code. Hereinafter, in the present embodiment, the case where the source code is described in Python as an example will be described, but the source code may be described in a programming language other than Python. Details of Phython are disclosed at https: / / www.python.org / . The content disclosed at this URL is incorporated herein by reference in its entirety.

[0048] First, developers or the like create the source code illustrated in FIG. 13 using a text editor or the like. The acquisition unit 110 (see FIG. 12) of the learning device 10 acquires the thus created source code and stores it in the storage unit 120. Next, the execution unit 130 executes each code included in the source code stored in the storage unit 120 line by line. When the source code does not include control statements such as if statements and for statements as illustrated in FIG. 13, the execution unit 130 sequentially executes each line from the first line to the last line in order from top to bottom. Conversely, when the source code includes control statements, the execution unit 130 executes each code in the order according to the control statements.

[0049] The content of the source code illustrated in FIG. 13 will be described. Lines 1 to 3 describe the registration of Functions including parameters by FunctionSet. Specifically, here, Functions including weights (in this example, instances l1, l2, l3 of the Linear class, which is a Function subclass that defines a layer algorithm for performing an inner product) are registered in an object of the class FunctionSet. The Function including weights can have its weights updated by an Optimizer. FunctionSet is a mechanism for improving code readability by grouping Functions updated by an Optimizer together.

[0050] Lines 4 and 5 describe the initialization of the Optimizer. In line 4, an instance of a subclass of Optimizer (a class for updating weights), which implements the Adam algorithm, is generated. The processing content of Adam is to execute the update by the formula described in the above "2-13" for each element of the weights. In line 5, a list of Functions including the weights defined in lines 1 to 3 is passed to the setup method of the instance of the Optimizer subclass generated in line 4. By executing this setup method, the internal state of the Optimizer subclass for updating the weights included in the list of Functions passed to this method is initialized.

[0051] Line 6 describes the loading of the input data. That is, line 6 exemplifies the process of reading the input data x and t from a file or the like. In this example, x holds data with a large amount of information such as images and voices, and t holds the label ID corresponding to x (data with a small amount of information for answer verification).

[0052] Line 7 describes the holding of the input data by a Variable object. That is, in line 7, an object of the Variable class for holding the input data is generated. The "Define-by-Run" function is realized by the mutual dependence of the Variable object and the Function object, and since arbitrary input data does not have a mechanism for realizing the "Define-by-Run" function, a procedure for explicitly holding it by an instance of the Variable class is required.

[0053] Lines 8 to 11 describe the execution of the forward process. Specifically, in Lines 8 to 11, the Forward process is executed by the description in a general programming language. By the "Define-by-Run" function, a reference structure for the backward process is generated simultaneously with the execution of this definition. By the mutual reference between instances of the Function class and instances of the Variable class, the correspondence between arbitrary processes and data can be expressed. Since the Variable class represents data and the Function class represents processes, this is self-evident. A data structure that represents the backward calculation procedure shown in Figure 2 using this reference structure is defined as the reference structure for the backward process. The reference structure for the backward process grows each time a basic calculation (arithmetic operations and exponentiation) on a Variable object and a call to a Function that takes a Variable object as an argument or return value are performed. Therefore, even in the description of the forward process that includes function calls other than those for branches, loops, Functions, and Variables, a reference structure for the backward process can be generated. There are also corresponding Function subclasses for basic calculations on Variable objects respectively.

[0054] Line 12 describes the execution of the backward process. Specifically, Line 12 executes the backward process by calling the backward method of the loss variable (an instance of the Variable class) obtained as the execution result of the forward process executed in Lines 8 to 11. The backward process is automatically executed in the reverse order of the forward process by following the reference structure for the backward process generated during the execution of the forward process.

[0055] Line 13 describes the weight update. Specifically, in line 13, the gradient of the weights is calculated as a result of performing the backward process in line 12. When the update method of the instance of the Optimizer subclass is called as in line 13, the weights are updated using this weight gradient. Since the call to the update method for the Optimizer subclass and the call to the backward method of the Variable class are separate functions, it is also possible to perform the weight update after partially executing the backward process. This is effective when you do not want to update the weights for the already learned Function.

[0056] Here, as the content processed during the forward process, pay particular attention to the content processed by the code described in line 8 in particular. Line 8 is described as h1 = F.relu(model.l1(x)).

[0057] When "model.l1(x)" is executed, the following reference structure for the backward process is generated.

Number

[0058] The above reference structure becomes a reference structure for backward processing as follows after the execution of "F.relu(".

Number

[0059] For the sake of caution, the reason for inserting the Splitter into the reference structure will be explained. If "model.l1(x)" is called again immediately after creating the above reference structure, the following reference structure is generated.

Number

[0060] When propagating the error value from the output layer during backward processing, set the error of x to the result of adding and combining the error values passed to x' and x'' respectively by the instance of splitter. By inserting the splitter in this way, the error can be propagated from all Functions that used x as an input during forward processing during backward processing.

[0061] Next, supplement the network configuration of the neural network generated when the source code illustrated in FIG. 13 is executed. FIG. 14 is a schematic diagram conceptually showing the network configuration of the neural network generated by the source code shown in FIG. 13. In FIG. 14, the blocks drawn with dotted lines indicate instances of variables, and the blocks drawn with solid lines indicate functions.

[0062] First, when the 7th line is executed, an instance 30 of variable x and an instance of variable t are generated. For convenience of explanation, only the instance 30 of variable x is shown in FIG. 14, but in reality, an instance of variable t is also generated in the same way. When the 7th line is executed, the instance of variable x actually holds data such as images and sounds.

[0063] Next, when the 8th line is executed by the execution unit 130, a neural network in a state where, after the instance 30 of the variable x, the function "l1" 31, the function "relu" 32, and the instance 33 of the variable h1 grow sequentially is generated. Note that when the 8th line is executed, the execution result of the forward process described in the 8th line is already held by the instance 33 of the variable h1. Also, when the 8th line is executed, as described above, a reference structure for the backward process being generated at the current time is generated.

[0064] Next, when the 9th line is executed by the execution unit 130, a neural network in a state where, after the instance 33 of the variable h1, the function "l2" 34, the function "relu" 35, and the instance 36 of the variable h2 grow sequentially is generated. Note that when the 9th line is executed, the execution result of the forward process described in the 9th line is already held by the instance 36 of the variable h2. Also, when the 9th line is executed, as described above, a reference structure for the backward process being generated at the current time is generated.

[0065] Similarly, when the 10th line is executed by the execution unit 130, a neural network in a state where, after the instance 36 of the variable h2, the function "l3" 37 and the instance 38 of the variable y grow sequentially is generated. Note that when the 10th line is executed, the execution result of the forward process described in the 10th line is already held by the instance 38 of the variable y. Also, when the 10th line is executed, as described above, a reference structure for the backward process being generated at the current time is generated.

[0066] Finally, when the 11th line is executed by the execution unit 130, a neural network in which the function "Softmax" 39 and the instance 40 of the variable loss grow sequentially after the instance 38 of the variable y is generated. Note that when the 11th line is executed, the execution result of the forward process described in the 11th line is already held in the new instance 40 of the variable loss. Also, when the 11th line is executed, as described above, a reference structure for the backward process currently being generated is generated. At the time when the 11th line is executed, the forward process described in the source code is completed. That is, at the time when the 11th line is executed, the difference between the identification result obtained by the finally obtained neural network and the true identification result given by the variable t is held in the instance 40 of the variable loss. Using this difference as an input, the backward process of the next step is executed.

[0067] After the forward process described in lines 8 to 11 is completed, next, when the 12th line is executed by the execution unit 130, the backward process is executed. Since the reference structure for the generated backward process has already been generated, the execution unit 130 can calculate the weight gradients of each intermediate layer (only the intermediate layers with weights) included in the neural network by executing the backward process based on this reference structure.

[0068] Next, the 13th line is executed by the execution unit 130. As a result, using the weight gradients calculated by the execution of the 12th line, the weights of each intermediate layer (only the intermediate layers with weights) are updated. That is, learning is executed.

[0069] Thus, in the learning device according to this embodiment, for forward processing, developers or the like can construct a neural network by describing line by line which variable instance's execution result obtained by providing an instance of any variable to any function is to be held by which variable instance. As a result, developers or the like can easily and intuitively describe forward processing in the source code. Also, developers or the like can describe forward processing in the source code (without having to be conscious of backward processing), and by causing the learning device according to this embodiment to execute the source code, the learning device can be caused to automatically execute backward processing.

[0070] 6. Comparative Example 1 Next, to show the superiority of the learning device according to this embodiment, a case where processing equivalent to that executed by the source code illustrated in FIG. 13 is described using Caffe according to the prior art will be explained. FIG. 15 is a diagram showing an example of the source code described using Caffe according to the prior art.

[0071] As shown in FIG. 15, the definition of a layer (corresponding to Function in this embodiment) is described in a block surrounded by {} described immediately after the term "layer". In this method according to the prior art, it is necessary to explicitly indicate the dependency between layers in the code. For example, the descriptions "top" and "bottom" represent the dependency between layers. "Bottom" represents from which layer the input to the layer is obtained, and "top" represents to which layer the processing result at the layer is output.

[0072] In this method according to the prior art, it is necessary to statically define the network configuration prior to the learning and identification processing performed by the neural network. That is, it is necessary to first define the network configuration of the neural network and then execute the learning and identification of the neural network. Therefore, it is difficult to dynamically change the network configuration according to the nature of the data. In contrast, in the learning device according to the present embodiment, as described above with reference to FIG. 14, when each code defining the configuration of the neural network is executed, the forward process corresponding to the code is executed. That is, the definition of the configuration of the neural network and the execution of the forward process according to the configuration are executed at the same timing. Thereby, it is also possible to easily change the configuration of the network according to the nature of the data. For example, a branch may be added to the code in FIG. 13, and the layer for executing the forward process may be switched according to the value of the variable t or the data size of the variable x. Also, for example, a variable value can be given as input data instead of the constant "10" in the 9th line of the code in FIG. 19.

[0073] In addition, in the method according to the prior art, when creating the source code, a developer or the like needs to describe the definition of the network configuration of the neural network so that both the forward process and the backward process can be appropriately executed. In contrast, in the learning device according to the present embodiment, as described above with reference to FIG. 14, without the need to be aware of whether the backward process can be appropriately executed, after simply describing the forward process (network configuration), the source code is caused to be executed by the learning device, and the learning device automatically executes the backward process. Therefore, a developer or the like can easily and efficiently construct a neural network and execute identification and learning.

[0074] Furthermore, in the method according to the prior art, when creating the source code, a developer or the like follows the procedure of defining the neural network so that both the forward process and the backward process can be appropriately executed, and then substituting data (input data, teacher data, etc.) into the thus-defined neural network. Therefore, it is difficult to describe the source code intuitively. On the other hand, in the learning device according to the present embodiment, when developers or the like describe the forward process (network configuration) line by line, they describe the source code in a manner of specifying line by line which variable instance should hold the execution result obtained by providing the instance of which variable to which function. As a result, developers or the like can describe the source code intuitively.

[0075] 7. Operation Example 2 Next, another specific example of the operation performed by the learning device 10 according to the embodiment having the above configuration will be described. FIG. 16 is a diagram showing another example of the source code input to the learning device according to an embodiment of the present invention. Note that it should be noted that the source code illustrated in FIG. 16 is intentionally simplified for the purpose of explaining the features of the learning device according to the present embodiment. Also, the line numbers described at the left end in FIG. 16 are added for explaining this specific example and are not included in the actual source code.

[0076] With reference to the source code shown in FIG. 16, it will be described that according to the learning device according to the present embodiment, it is also possible to easily construct a neural network using a control statement (here, a for statement).

[0077] Regarding the first to third lines, since they are the same as the first to fifth lines in the source code shown in FIG. 13, detailed description thereof will be omitted. The fourth line describes that the processing described in the fifth to tenth lines is looped until the value of i becomes 0 to 1000. Regarding the fifth and sixth lines, since they are the same as the sixth and seventh lines in the source code shown in FIG. 13, detailed description thereof will be omitted. The seventh line describes that y, which is the processing result of the functions l1 and relu, is added again as an argument of l1. Regarding the eighth to tenth lines, since they are the same as the eleventh to thirteenth lines in the source code shown in FIG. 13, detailed description thereof will be omitted.

[0078] FIG. 17 is a schematic diagram conceptually showing the network configuration of a neural network generated by the source code shown in FIG. 16. In FIG. 17, the blocks drawn with dotted lines indicate instances of variables, and the blocks drawn with solid lines indicate functions. Further, FIG. 17 shows only the configuration of the neural network generated only when the variable i is 0 to 2 for convenience of explanation.

[0079] As is clear from FIGS. 16 and 17, in the learning device according to the present embodiment, even a neural network that repeats a plurality of the same configurations including variable instances and functions (here, after the function 52 that adds the instance 51 of the variable x and the instance 50 of the variable y, the functions "l1" 53 and "relu" 54 are sequentially continued, and the output value of the function "relu" 54 is held in the instance of the variable y) can be easily constructed using a simple control statement (here, a for statement). That is, it can be seen that the source code used in the learning device according to the present embodiment has a high affinity with the control statements of the programming language.

[0080] 8. Comparative Example 2 Next, in order to show the superiority of the learning device according to the present embodiment, a case where processing equivalent to that executed by the source code illustrated in FIG. 16 is described using Caffe according to the prior art will be described. FIG. 18 is a schematic diagram conceptually showing the network configuration of a neural network generated by the source code described using Caffe according to the prior art.

[0081] When attempting to construct a neural network similar to that illustrated in FIGS. 16 and 17 using Caffe according to the prior art, since the configuration of the neural network cannot be defined using control statements, developers etc. first define a basic configuration as shown in FIG. 18. Next, developers etc. must specifically describe the process of giving an initial value of the instance 75 of the variable y to the function 72 and giving the instance 75 of the variable y at the previous time and the instance 71 of the variable x at the current time to the function 72 (the portion of the arrow drawn with a thick line in FIG. 18). When constructing a neural network in which the above basic configuration is repeatedly performed many times, or when constructing a neural network having a multi-layer structure, developers etc. must make such a special description each time such a repeated operation is performed, or for each layer in the multi-layer structure.

[0082] In contrast, in the learning device according to the present embodiment, as illustrated in FIGS. 16 and 17, the source code to be described can be easily described without requiring special description using the control statements of the programming language. Therefore, according to the learning device according to the present embodiment, even a complex or large-scale neural network can be constructed easily and efficiently.

[0083] Regarding the additional function (1) The learning device according to one embodiment may be capable of executing a function that cuts off the reference structure for the backward process. Specifically, when the unchain_backward method of the instance of the Variable class is called, the reference structure for the backward process going toward the input side starting from that instance is cut off. For example, assume that the following reference structure for the backward process has been generated by the execution of the forward process (detailed configurations such as splitters are shown in an omitted manner).

[0084] A (input layer) ← Convolution2D ← B ← Linear ← C ← Softmax ← D (output layer) Here, A, B, C, and D represent instances of the Variable class, and Convolution2D, Linear, and Softmax represent instances of the Function class.

[0085] At this time, when calling B.unchain_backward(), the reference structure for the backward process is cut off starting from B, resulting in the following changes. B ← Linear ← C ← Softmax ← D (output layer)

[0086] Consider the situation of applying this unchain_backward method to the source code shown in FIG. 16. In this source code, at line 7, y, which is the processing result of function l1 and function relu, is added back to the argument of function l1 again. In the "Define-by-Run" mechanism, when executing the description of "x + y", a copy of y is generated, and the reference structure for the backward process generated by the forward process executed so far is concatenated. Therefore, in this example, the reference structure for the backward process continues to grow each time the loop is repeated. The backward process executed at line 9 is executed on the grown reference structure for the backward process. Since the process at line 9 is included in the loop, the total calculation time of this loop process is proportional to the square of the loop size.

[0087] FIG. 19 is a diagram showing still another example of the source code input to the learning device according to an embodiment of the present invention. Note that the line numbers described at the left end in FIG. 19 are attached for explaining this specific example and are not included in the actual source code.

[0088] By modifying the source code shown in FIG. 16 and periodically calling unchain_backward at line 11 as shown in FIG. 19, an increase in the calculation time can be suppressed. Line 9 describes that the processes of lines 10 to 12 are executed each time the loop from line 4 onwards is executed 10 times. Line 11 calls unchain_backward and discards the reference structure for backward processing starting from the loss. This can reduce the overall computation time of the loop processing.

[0089] By using unchain_backward in this way, even when learning is performed for forward processing with a loop in the reference structure, excessive growth of the reference structure for backward processing can be suppressed, and the learning process can be executed with a realistic computational load. Furthermore, in another embodiment, it is also possible to use unchain_backward for the purpose of not updating the weights for a specific Function.

[0090] 10. Regarding Additional Function (2) The learning device according to one embodiment can specify the volatile attribute during the initialization of an instance of the Variable class. When the volatile attribute is valid, no reference structure for backward processing is generated for the forward processing that takes this Variable as input.

[0091] When only forward processing is executed using learned weights (i.e., when backward processing is not required), if the process of generating the reference structure for backward processing is executed during the execution of forward processing, waste will occur in both the execution speed and the memory usage. In such a case, by specifying the volatile attribute during the initialization of an instance of the Variable class that holds the input data for forward processing, the generation of the reference structure for backward processing can be stopped, and only forward processing can be efficiently executed.

[0092] 11. Postscript As the most preferred embodiment, the embodiment in which the source code described in Python is input to the learning device has been described. However, the technology disclosed in this specification is not limited to the case where the source code described in Python is used. That is, the technology disclosed in this specification includes that when the learning device executes each code, it calculates the output value of the forward process described in the code based on the input value, that when the learning device executes the forward process described in each code, it generates a reference structure for the backward process (so that the backward process can be executed based on this reference structure), and that it defines the configuration of the neural network using control statements. The technology is similarly applicable when using source code described in a programming language equivalent to Python (for example, R, Julia, Sparkz, and MLib, etc.) that can realize at least one of the above.

[0093] The technology disclosed in this specification can be realized by executing the source code described in Python and a programming language equivalent thereto. Alternatively, it may be realized by executing a module or library described in Python and a programming language equivalent thereto.

[0094] In this specification, the names used to identify variables, functions, methods, classes, subclasses, etc. do not limit the technology disclosed in this specification and may be arbitrary.

[0095] The processes and procedures described in this specification can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination thereof. Specifically, the processes and procedures described in this specification are realized by implementing the logic corresponding to the processes in media such as integrated circuits, volatile memories, non-volatile memories, magnetic disks, and optical storage. Also, the processes and procedures described in this specification can be implemented as a computer program and executed on various computers.

[0096] Even if it is described that the processes and procedures described in this specification are executed by a single device, software, component, or module, such processes or procedures can be considered to be executed by a plurality of devices, a plurality of software, a plurality of components, and / or a plurality of modules. Also, even if it is described that the data, tables, or databases described in this specification are stored in a single memory, such data, tables, or databases can be considered to be stored distributively in a plurality of memories provided in a single device or a plurality of memories arranged distributively in a plurality of devices. Furthermore, the software and hardware elements described in this specification can be considered to be realized by integrating them into fewer components or decomposing them into more components.

[0097] Part 2 (Implementation Method for Embedding Algorithm in Chip) 1. Background Deep learning is an algorithm that, while achieving high performance, requires a large amount of computation, memory usage, and learning sample volume. The spread of GPUs and clouds that can obtain abundant computing resources at low cost, and the Web infrastructure that enables sharing of learning samples, can be said to be the background that has supported the recent prosperity of deep learning. There are various environments (libraries, frameworks) that support the development of deep learning algorithms. Many development environments have the function of improving the learning speed by using GPUs.

[0098] In fields such as fully autonomous driving of automobiles and highly versatile robot control, in order to analyze information acquired from various sensors such as cameras and LIDAR (laser distance measurement) in real time and control countless motors to solve the problems, advanced information processing capabilities are required. Therefore, the application of deep learning with performance that is a cut above the conventional is strongly expected. However, these fields rely on embedded environments that are lacking in computing resources compared to GPUs and the cloud due to requirements such as safety, chip price, and power consumption. Therefore, the application of deep learning that requires high computing resources has been delayed. The reasons for the delay in the application of deep learning to embedded environments include, in addition to the aspect that the computing resource requirements of these algorithms exceed the performance of realistic and economic embedded environments, the fact that implementations that support deep learning, starting with the software environment, are not yet available. Even in embedded environments, the performance of hardware is improving year by year, and in deep learning algorithms, improvements to relax the requirements for computing resources are continuing. Therefore, it is considered that the former factor will be gradually solved.

[0099] The problem to be solved by the embodiments of the present invention is to break through the barriers to the adaptation of deep learning to the embedded environment remaining mainly in the software environment aspect by developing a framework for designing a deep learning algorithm that operates while meeting product-level requirements in an embedded chip, and to accelerate the development speed. In the first part, it was described that the learning device according to the embodiment is a framework that provides high productivity in the development of deep learning algorithms while being GPU-based. Since functional expansion of the learning device according to the embodiment for an embedded environment is considered to be the most optimal means for solving this problem, the problems of adapting to the embedded environment focused on the learning device according to the embodiment will be described in the following paragraphs.

[0100] 2. Problems of the implementation method according to the embodiment Since the learning device according to the embodiment described in the first part depends on advanced language functions and libraries, attempting to directly operate the algorithms operating in this learning device on an embedded system semiconductor chip may cause the following adverse effects. First, regarding security, as the scale of libraries and languages increases, the degree to which applications depend on de facto unknowable implementations increases. Along with that, the risk that defects contained in such implementations will directly become defects in the chip product increases. Next, regarding the footprint, the implementation of libraries and languages themselves compresses the memory resources of the chip product. Furthermore, regarding overhead, via libraries with highly abstracted APIs, the computing resources of the chip product cannot be fully utilized. At least for the large-scale calculations required by neural networks, low-level performance tuning specialized for the chip is essential. For the reasons described above, simply operating the algorithms operating in the learning device according to the embodiment directly on an embedded system semiconductor chip is likely to fail to meet the product-level requirements.

[0101] 3. Concept of the implementation method according to the embodiment In the implementation method according to the embodiment, a new neural network (NN) algorithm designed on a personal computer or the like having abundant computing resources is realized in the shortest possible time in a state where it can operate while satisfying product-level requirements in any embedded chip (embedded semiconductor integrated circuit). For this purpose, it is desirable that the developer who designs the algorithm and the developer who is highly conscious of the hardware can work as independently as possible. In this embodiment, a technical idea regarding a device (framework) that assists this is proposed.

[0102] 4. Development steps assumed in embedded chip development When developing an embedded chip, the following three steps are assumed. Step I: A state in which the code (code written in Python as an example) used in the learning device according to the embodiment is running on a PC (+ GPU) This state is a state in which the design and verification of an algorithm using a neural network having a complex configuration are realized with less code description. This is the concept of the above-mentioned "Define-by-Run" method. Step II: A state in which an implementation optimized for the chip and Python code are mixed This state is a state in which the operation confirmation and performance verification of the algorithm designed in the learning device according to the embodiment on the chip are realized with little change to the Python code. Step III: A state in which the algorithm designed in the learning device according to the embodiment operates only with an implementation optimized for the chip This state is a state in which the algorithm operates while satisfying the product-level specification requirements as a chip (real-time cooperative operation with other modules and control mechanisms on the chip is possible). In the implementation method according to the present embodiment, when developing a new algorithm in the learning device according to the embodiment, a framework that can advance the development in a short period of time is proposed by saving the trouble of re-modification, re-design, and re-learning as much as possible between the steps of Steps I to III above.

[0103] 4-1. Regarding Step I FIG. 20 is a schematic diagram for explaining Step I of the implementation method according to an embodiment of the present invention. The configuration shown in FIG. 20 is premised on the configuration of the learning device according to the embodiment described in the above first part. That is, in this configuration, the source code written in Python as an aspect of the programming language uses PyCUDA as an aspect of the library and numpy (BLAS) as an aspect of the library, and these libraries drive the GPU and the general-purpose computer, respectively. Note that "Chainer" shown in FIG. 20 is a name given by the applicant of the present application to the framework for describing the source code used in the learning device according to the embodiment described in the above first part.

[0104] 4-2. Regarding Step II FIG. 21 is a schematic diagram for explaining Step II of the implementation method according to an embodiment of the present invention. In the configuration shown in FIG. 21, the front end of Chainer is executed on Python. As shown in FIG. 21, in this embodiment, by providing a Native I / F (an interface for calling an implementation equivalent to the main functions of Chainer described in a low-level language such as C language), it is possible to execute the execution on the PC and the execution optimized for the embedded chip with the same code.

[0105] FIG. 22 is a schematic diagram for explaining the case where the execution part by Python and the execution part by the chip communicate with each other. As shown in FIG. 22, by providing a communication function in the implementation of the Native I / F, it is also possible to remove the dependence on Python from the configuration on the embedded chip (drive the optimized implementation on the embedded chip from Chainer on the PC).

[0106] Regarding the implementation of Native I / F Implement reference codes (assuming a low-level language such as C) for Chainer's Function and Optimizer. Implement this reference code in a form that does not depend on external libraries such as numpy. Also, implement a memory pool mechanism suitable for dynamic network definitions. Additionally, create a data conversion function between numpy and Function / Optimizer through a separate interface. Furthermore, create a floating-point version reference code for the above Function / Optimizer. Furthermore, create a fixed-point version reference code for the above Function / Optimizer. Also, create a data conversion function between floating-point and fixed-point through a separate interface from Function / Optimizer. This is because there are still many chips without an FPU. Based on the above reference code, implement codes optimized for various chips.

[0107] 4-3. Regarding Step III FIG. 23 is a schematic diagram for explaining Step III of the implementation method according to an embodiment of the present invention. As shown in FIG. 23, a method for outputting a network definition and weights as bytecode from Chainer is added. Also, a virtual machine for interpreting the bytecode and executing neural network processing (forward processing, backward processing, weight update) is provided. Optimization implementation for the Native I / F chip can be reused.

[0108] Configuration 1 (Configuration of Native IF) FIG. 42 is a diagram showing a configuration example of Native I / F according to an embodiment of the present invention. For each NN algorithm, a configuration that provides an interface independent of the type of computer. A processing system using the NN algorithm instructs a specific computer to execute the algorithm via this interface. The interface mentioned here refers to the means for defining the input data format, output data format, and the correspondence between the processing method of the input data format and the output data format. If the interfaces are the same, the same output result can be obtained for the same input. For example, a function described in the C language and its function declaration can be cited. The processing system on the side using the NN algorithm is not particularly limited. For example, existing frameworks (such as Chainer) for NN design can be cited. Also, a processing system developed in conjunction with the development of the algorithm can be cited. The computer mentioned here means a device that executes calculations. A computer is a device that includes an arithmetic core, a memory hierarchy, and the hardware resources necessary to execute calculations. A general-purpose computer means a commonly used computer. It is a computer on which conventional algorithms including the Linux (registered trademark) OS and Python can operate easily. The accelerator mentioned here means a device that executes specific calculations including the calculations of the NN algorithm at high speed. The GPU mentioned here is a computer specialized for image processing but also has the ability to execute general-purpose calculations. The GPU also includes one form of the accelerator mentioned above. Because there are software assets such as CUDA, the ease of implementing the NN algorithm is at an intermediate level between that of a general-purpose computer and that of a general accelerator.

[0109] Configuration 1-1 (Configuration for executing identification and learning by NN) FIG. 43 is a diagram showing a configuration example for executing identification and learning by NN according to an embodiment of the present invention. The Native I / F has at least a Forward processing unit. With this configuration, the Native I / F can execute identification processing using the NN algorithm. Furthermore, the Native I / F has at least a Forward processing unit, a Backward processing unit, an internal state initialization processing unit for the weight update algorithm, and a weight update processing unit. With such a configuration, the Native I / F can execute identification processing and learning processing using the NN algorithm. The Forward processing unit and the Backward processing unit are included for each layer algorithm. The internal state initialization processing unit for the weight update algorithm and the weight update processing unit are included for each weight update algorithm. Furthermore, the Native I / F has, for each layer algorithm, a Forward processing call interface and a Backward processing call interface, and, for each weight update algorithm, an internal state initialization processing interface for the weight update algorithm and a weight update processing call interface. Furthermore, the implementation on the side called through the Native I / F has a NativeI / F call management unit. With such a configuration, the implementation on the side called through the Native I / F can change the implementation that can optimally execute the operation of the Native I / F according to the difference in the parameters of the Native I / F. Incidentally, if there is no implementation that can execute the operation of the Native I / F, the call management unit of the Native I / F returns an error to the caller. Therefore, the implementation on the side called through the Native I / F can select and execute the implementation that can optimally execute its operation.

[0110] Configuration 1-1-1 (Configuration 1 for executing identification / learning by NN; in the case of a configuration (multi-dimensional array management unit) for managing multi-dimensional arrays) FIG. 44 is a diagram showing a configuration example for managing a multi-dimensional array according to an embodiment of the present invention. The Native I / F further includes a multi-dimensional array management unit. The multi-dimensional array management unit can perform at least one selected from the group including generation, destruction, acquisition of attributes (number of axes, number of elements per axis), acquisition of aggregation results (sum, average, variance, etc. per axis), and arithmetic operations on a per-element basis between multi-dimensional arrays.

[0111] Configuration 1-2 (Configuration for sharing data)

[0112] Configuration 1-2-1 (Configuration 1 for sharing data; in the case of a data representation conversion unit) FIG. 45 is a diagram showing a configuration example of a data representation conversion unit according to an embodiment of the present invention. Furthermore, the Native I / F has a data representation conversion unit. The data representation conversion unit can mutually convert a data representation (device-dependent data representation) that depends on a specific computer and a data representation (device-independent data representation) that does not depend on a specific computer in the Native I / F.

[0113] Configuration 1-2-2 (Configuration 2 for sharing data; when having an external storage medium) Furthermore, the processing system on the side that calls the Native I / F has an external storage medium. The external storage medium can store weight data converted into device-independent data.

[0114] Configuration 1-2-3 (Configuration 3 for sharing data; when having a communication unit) FIG. 46 is a diagram showing a configuration example of a communication unit according to an embodiment of the present invention. Furthermore, the implementation on the called side through the Native I / F has a communication unit. The communication unit can communicate the call information of the Native I / F to the implementation on the called side. Also, when any processing system using the NN algorithm attempts to call the Native I / F regardless of the presence or absence of communication of call information, the implementation on the side called through the Native I / F can execute optimal communication processing as needed. By this process, the physical distance of the computer, the presence or absence of memory sharing, or the difference in communication protocols can be hidden from any processing system using the NN algorithm. For example, Native I / F regardless of the presence or absence of communication of call information includes an interface for executing a layer algorithm, an interface for executing a weight update algorithm, or an interface for executing data representation conversion.

[0115] Configuration 2 (Configuration of the extended version of Native I / F)

[0116] Configuration 2-1 (Configuration 1 of the extended version of Native I / F; when having a type conversion unit, and an NN algorithm execution unit for floating point and / or an NN algorithm execution unit for fixed point) FIG. 47 is a diagram showing a configuration example of an execution unit and a type conversion unit for floating point and fixed point according to an embodiment of the present invention. The Native I / F has a type conversion unit, and an NN algorithm execution unit for floating point and / or an NN algorithm execution unit for fixed point. For example, there are computer B having only a type conversion unit, computer A having only an NN algorithm execution unit for floating point numbers, or computer C having only an NN algorithm execution unit for fixed point numbers. When such computer A, computer B, and computer C are combined with the basic configuration of the Native I / F, the floating point type data generated by computer A is transferred to computer B. Subsequently, the data transferred from computer A to computer B is converted into fixed point type data by computer B. Then, the fixed point type data converted by computer B is transferred to computer C. And the fixed point type data transferred from computer B becomes the input data of computer C, and the overall operation of the NN algorithm is executed. Such a process can also be executed in reverse order.

[0117] Configuration 2-2 (Configuration 2 of the extended version of Native I / F; when having a memory pool module) FIG. 48 is a diagram showing a configuration example of a memory pool according to an embodiment of the present invention. Furthermore, the implementation on the called side through the Native I / F has a memory pool module. The memory pool module can realize dynamic memory management.

[0118] Configuration 2-3 (Configuration 3 of the extended version of Native I / F; when having an algorithm execution unit that fuses multiple NN algorithms) FIG. 49 is a diagram showing a configuration example of an algorithm execution unit that fuses multiple NN algorithms according to an embodiment of the present invention. Furthermore, the Native I / F has an algorithm execution unit that fuses multiple NN algorithms. The algorithm execution unit that fuses multiple NN algorithms simultaneously executes multiple algorithms for frequently occurring combinations of NN algorithms.

[0119] Configuration 2-4 (Configuration 4 of the extended version of Native I / F; when having a multi-dimensional array data compression / decompression unit) FIG. 50 is a diagram showing a configuration example of a multi-dimensional array data communication volume reduction unit according to an embodiment of the present invention. Furthermore, the implementation on the side called through the Native I / F has a multi-dimensional array data compression / decompression unit. The multi-dimensional array data compression / decompression unit is provided in the communication unit.

[0120] Configuration 3 (Configuration of Native I / F + Chainer Execution Unit) FIG. 51 is a diagram showing an example of cooperation with an existing execution unit according to an embodiment of the present invention.

[0121] Configuration 3-1 (Configuration 1 of Native I / F + Chainer Execution Unit; when having a bytecode generation unit and a virtual machine) FIG. 53 is a diagram showing a configuration example of a bytecode generation unit and a virtual machine according to an embodiment of the present invention. Furthermore, the Chainer execution unit has a bytecode generation unit. The bytecode generation unit inputs the calculation procedure and weights of Backward and outputs them as bytecode. For example, the bytecode generation unit is provided in the Python layer of Chainer. Also, the Native I / F has a virtual machine. The virtual machine interprets the bytecode and executes the NN algorithm process. The NN algorithm process referred to here is any one of, or a combination of, forward processing, backward processing, and weight update.

[0122] Configuration 3-2 (Configuration 2 of Native I / F + Chainer Execution Unit; when having a comparison unit) FIG. 54 is a diagram showing a configuration example of a comparison unit according to an embodiment of the present invention. Furthermore, the Chainer execution unit has a comparison unit. The comparison unit compares the input / output results of the existing execution unit and the Native layer execution unit corresponding to the same NN algorithm, or compares the input / output results of the Native layer execution units of different implementations called in the same Native I / F.

[0123] Configuration 3-3 (Configuration of Native I / F + Chainer Execution Unit; when having a function synthesis unit) FIG. 55 is a diagram showing a configuration example of a function synthesis unit according to an embodiment of the present invention. Furthermore, the Chainer execution unit has a function synthesis unit. The function synthesis unit inputs the calculation procedure of Backward and replaces a combination of instances of Function classes that can support the "Native I / F for simultaneously executing multiple algorithms" with an instance of a Function class corresponding to the "Native I / F for simultaneously executing multiple algorithms". However, in the implementation of the Native layer for a computer that executes the calculation procedure of Backward, if there is no "Native I / F for simultaneously executing multiple algorithms", the above replacement is not performed. The replacement mentioned here can be executed by partial match search when the calculation procedure of Backward is regarded as a character string. For example, the function synthesis unit is provided in the Python layer of Chainer.

[0124] Configuration 4 (Configuration of an optimization device specialized for Forward processing execution)

[0125] Configuration 4-1 (Configuration 1 of an optimization device specialized for Forward processing execution; when having weight optimization processing means) Furthermore, the Chainer execution unit has weight optimization processing means. The weight optimization processing means executes a weight optimization process suitable for the Function class.

[0126] Configuration 4-2 (Configuration 2 of an optimization device specialized for Forward processing execution; when having means for reusing the data memory area) Furthermore, the Chainer execution unit and the Native I / F have means for reusing the data memory area. The means for reusing the data memory area reuses the memory area of the data input and output between layers. The reuse means is provided in the Forward processing execution unit or the virtual machine. For example, a flag for identifying that only the Forward processing is to be executed is provided as an argument of the interface (defined in the NativeI / F) that executes the Forward processing of the virtual machine. The condition for executing this process is when the volatile attribute is specified for the Variable variable input by the instance of the Function class of Chainer, or when the flag for identifying that only the Forward processing is to be executed is valid when the Forward processing of the virtual machine is executed.

[0127] Function 1 (Function of the NativeIF configuration) It becomes easier to divide the work between developers who design and use the NN algorithm and developers who are deeply aware of the hardware configuration of the computer. For example, since the identity of the interface for each NN algorithm to be executed is guaranteed by the Native I / F, developers who design and use the algorithm can execute the process on various computers without changing the software on the calling side of the Native I / F. Specifically, the risk of depending on the software developed by oneself for a specific computer can be reduced. As a result, it becomes possible to select a computer based on more essential criteria such as the price of the computer and its strengths and weaknesses for specific applications. For developers who are deeply aware of the hardware configuration of the computer, if an implementation optimized for a computer compatible with the Native I / F is provided, users of a wide range of NN algorithms can be made to use the computer developed by oneself.

[0128] Function 1-1 (Function of the configuration for executing identification and learning by NN) Developers who design and utilize the NN algorithm can realize the overall operation of the NN algorithm by calling the interfaces provided in the NativeI / F using any processing system that utilizes the NN algorithm. In addition, developers who design and utilize the NN algorithm can realize the overall operation of the NN algorithm using an implementation optimized for the computer being used without being aware of the specific configuration of the computer.

[0129] Effect 1-1-1 (Effect by Configuration 1 for Executing Identification / Learning by NN; in the case of a configuration for managing a multi-dimensional array (multi-dimensional array management unit)) When executing the overall operation of the NN algorithm, developers who design and utilize the NN algorithm can execute a combination of arbitrary NN algorithms without going through unnecessary data conversion processing. At this time, by checking the aggregation result of the contents of the multi-dimensional array, which is the processing result of an arbitrary NN algorithm, it is possible to check whether the NN algorithm is performing the intended calculations.

[0130] Effect 1-2 (Effect by Configuration for Sharing Data) Effect 1-2-1 (Effect by Configuration 1 for Sharing Data; in the case of a data representation conversion unit) By passing through the device-independent data representation, it is possible to exchange the data necessary to realize the overall operation of the NN algorithm between computers with different hardware configurations.

[0131] Effect 1-2-2 It is possible to hide the information specific to each computer. (Configuration 2 for Sharing Data; + when having an external storage medium) After converting the weight data into the device-independent data representation and storing it in the external storage medium, it is possible to execute the identification process on any computer using the learned weights on a specific computer.

[0132] Function 1-2-3 (Function by Configuration 3 for Sharing Data; when having a communication unit) Regardless of the hardware configuration of the computer, the physical distance, or the presence or absence of memory sharing, it is possible to exchange the data necessary to realize the overall operation of the NN algorithm. It is also possible to call the NN algorithm implementation installed on a computer where the processing system using the NN algorithm is inoperable from a computer where the processing system using the NN algorithm is operable. Therefore, it is possible to realize the overall operation of the NN algorithm using a plurality of computers connected to a computer network.

[0133] Function 2 (Function by the configuration of the extended Native I / F)

[0134] Function 2-1 (Function by Configuration 1 of the extended Native I / F; when having a type conversion unit, and an NN algorithm execution unit for floating point numbers, and / or an NN algorithm execution unit for fixed point numbers) In a hardware configuration where computers without a floating point unit (FPU) and computers with an FPU are mixed, it is possible to realize the overall operation of the NN algorithm using data types suitable for each computer. It is possible to realize the overall operation of the NN algorithm using floating point operations or fixed point operations. Specifically, computer A transfers floating point type data generated by the NN algorithm execution unit for floating point numbers of computer A to computer B. Next, computer B converts the transferred floating point type data from computer A into fixed point type data by the type conversion unit, and then transfers the fixed point type data to computer C. Computer C transfers the fixed-point data generated by the NN algorithm execution unit for fixed-point of computer C to computer B. Next, computer B converts the transferred fixed-point data from computer C into floating-point data by the type conversion unit, and then transfers the floating-point data to computer A.

[0135] Effect 2-2 (Effect by Configuration 2 of the Extended Version of Native I / F; When Having a Memory Pool Module) When a processing system that depends on the dynamic memory management mechanism calls the Native I / F including data generation and destruction and executes the entire operation of the NN algorithm, its operation can be realized in a lightweight manner.

[0136] Effect 2-3 (Effect by Configuration 3 of the Extended Version of Native I / F; When Having an Algorithm Execution Unit that Fuses Multiple NN Algorithms) Unnecessary access to global memory can be avoided. Also, the overhead of function calls can be reduced. Therefore, frequently occurring combinations of NN algorithms can be executed at high speed.

[0137] Effect 2-4 (Effect by Configuration 4 of the Extended Version of Native I / F; When Having a Multidimensional Array Data Compression / Decompression Unit) When executing the entire operation of the NN algorithm using multiple computers connected to a computer network, the data communication volume of the multidimensional array can be reduced. Therefore, the operation speed can be improved.

[0138] Effect 3 (Effect by Native I / F + Chainer Execution Unit) The entire operation of the NN can be defined and executed by combining an NN algorithm supported by Native I / F and an NN algorithm not supported by Native I / F. Once support for the Native I / F is obtained, the overall operation of the NN can be executed by appropriately replacing it with the Native I / F as needed. Therefore, it is not necessary to modify the existing software. Even when combining the Native I / F, the existing benefits of Define-by-run can be enjoyed.

[0139] Effect 3-1 (Effect by Configuration 1 of Native I / F + Chainer Execution Unit; when having a bytecode generation unit and a virtual machine) Since the Chainer execution unit has a bytecode generation unit and the Native I / F has a virtual machine, the dependency on advanced libraries and programming languages can be reduced. Therefore, in various computers including poor execution environments such as accelerators, the overall operation of the NN designed with Chainer can be executed while meeting the product-level requirements.

[0140] Effect 3-2 (Effect by Configuration 2 of Native I / F + Chainer Execution Unit; when having a comparison unit) The comparison unit compares the input / output results of the existing execution unit and the Native layer execution unit corresponding to the same NN algorithm, and also compares the input / output results of the Native layer execution units that call different implementations of the Native layer in the same Native I / F. By having such a comparison unit, the accuracy of the processing results of the NN algorithm execution unit for floating-point numbers and the accuracy of the processing results of the NN algorithm execution unit for fixed-point numbers can be compared. Therefore, it is possible to compare the processing results of the execution unit for which it has already been sufficiently tested that the NN algorithm can be correctly calculated, with the processing results of the newly created Native layer. Thus, it can be guaranteed that the implementation of the newly created Native layer can correctly calculate the NN algorithm.

[0141] Effect 3-3 (Operation according to Configuration 3 of Native I / F + Chainer Execution Unit; in the case of having a function synthesis unit) The function synthesis unit inputs the Backward calculation procedure and replaces a combination of instances of Function classes that the "Native I / F that executes multiple algorithms simultaneously" can handle with an instance of a Function class that has a 1:1 correspondence with the "Native I / F that executes multiple algorithms simultaneously". Note that if the "Native I / F that executes multiple algorithms simultaneously" does not exist, such a function synthesis unit does not perform the above replacement. By having such a function synthesis unit in the Python layer configuration of Chainer, the Backward calculation procedure is automatically processed regardless of the presence or absence of the "Native I / F that executes multiple algorithms simultaneously". By processing such a Backward calculation procedure, if the "Native I / F that executes multiple algorithms simultaneously" exists, the Native I / F is called and replaced with an instance of the corresponding Function class. As a result, the overall operation of the high-speed NN algorithm can always be realized. Also, even in a combination of functions where the "Native I / F that executes multiple algorithms simultaneously" does not exist, the function synthesis unit may exhibit merits. Specifically, it is a combination of Convolution2D + BatchNormalization or a combination of Linear + BatchNormalization when limited to Forward processing. BatchNormalization is a process that equalizes the variance and removes the mean for each element of the multi-dimensional array serving as its input, based on long-term statistical information obtained through the learning of the neural network. If only the Forward process is performed without learning, there is no need to update the variance or mean. For example, if a and b are constants, it is just a process of performing a transformation such as y = ax + b for each array element. The Linear process is a process of performing a matrix multiplication. Also, Convolution2D is a process of performing a combination calculation of convolution and matrix multiplication. Since these processes include a transformation such as y = ax + b exemplified above, by adjusting the weights and biases of Linear and Convolution2D, the same result as inputting and processing the output results of these Functions into BatchNormalization can be obtained. By performing such adjustments of weights and biases, the function synthesis unit can convert Convolution2D + BatchNormalization into a single Convolution2D. The same applies to the conversion from Linear + BatchNormalization to a single Linear.

[0142] Effect 4 (Effect by the configuration of the optimization device specialized for executing the Forward process) By reducing the amount of weight information in the Forward process or the data memory amount of the input data and executing the Forward process, the memory can be reduced. Also, by reducing the number of elements of the weights or by not calculating zero weights and executing the Forward process, the calculation amount can be reduced.

[0143] Effect 4-1 (Effect by the optimization device 1 specialized for executing the Forward process; when having a weight optimization processing means) By having the weight optimization processing means specialized for Forward processing execution in the Function class of Chainer, it is possible to execute weight optimization processing on any instance of the Function class included in the configuration of the learned network. In this way, by being able to execute weight optimization processing, it is possible to reduce the memory and computational amount during Forward processing. As a result, the overall operation of the NN algorithm can be executed at high speed.

[0144] Effect 4-2 (Operation by the optimization device 2 specialized for Forward processing execution; when having means for reusing the data memory area) By giving a flag for executing only Forward processing as an argument to the Forward processing execution unit (Chainer or virtual machine), it is possible to reduce the memory during Forward processing. As a result, the overall operation of the NN algorithm can be executed at high speed.

[0145] 5. Regarding the specific procedure of the implementation method according to the embodiment The implementation method according to the embodiment includes a first method and a second method. 5-1. Regarding the first method FIG. 24 is a schematic diagram showing a configuration example of an implementation device used in the implementation method (first method) according to an embodiment of the present invention. As shown in FIG. 24, the implementation device according to an embodiment mainly includes an evaluation board (mother board) 100 and an embedded chip (embedded semiconductor integrated circuit) 200 detachably mounted on the evaluation board 100.

[0146] The evaluation board 100 mainly includes a CPU 101, a main memory 102, a communication I / F 103, and an external memory 104. These components are electrically connected via an internal bus 109.

[0147] The CPU 101 loads various programs such as an operating system from the external memory 103 into the main memory 102 and executes the instructions included in the loaded programs. The main memory 102 is used to store the programs executed by the CPU 101 and is constituted by, for example, a DRAM.

[0148] The communication I / F 103 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination thereof, and is configured to be able to communicate with a computer (not shown) and input / output devices, etc. operated by a developer, etc. via a communication network (not shown) including Ethernet (registered trademark) and the Internet. Further, the communication I / F 103 can also communicate with the communication I / F 204 of the embedded chip 200 described later. The external memory 104 is constituted by, for example, a flash memory, etc., and stores various programs such as an operating system.

[0149] Next, the embedded chip 200 includes a CPU 201, an accelerator (auxiliary arithmetic unit) 202, a main memory 203, a communication I / F 204, and an external memory 205. These respective components are electrically connected via an internal bus 209. Note that the embedded chip can optionally include a GPU (not shown).

[0150] The CPU 201 loads the source code (for example, source code written in Python, etc.) received from the evaluation board 100 (via the communication I / F 103) via the communication I / F 204 into the main memory 203 and executes each code included in the loaded source code. The accelerator 202 loads the source code (e.g., source code described in C language, assembler, etc.) received from the evaluation board 100 (communication I / F 103 thereof) via the communication I / F 204 into the main memory 203, and executes each code included in the loaded source code. The main memory 203 is used to store the source code executed by the CPU 201 and the accelerator 202, and is configured by, for example, a DRAM.

[0151] The communication I / F 204 communicates with the communication I / F 103 of the evaluation board 100 to transmit and receive various information. The external memory 205 is configured by, for example, a flash memory or the like, and stores various data.

[0152] FIG. 25 is a flowchart showing an example of a procedure used in an implementation method according to an embodiment of the present invention. First, in step 301, the source code described in the first programming language (e.g., Python, etc.) is executed on a personal computer or the like. Developers and the like confirm whether or not the source code operates on a personal computer or the like based on the execution result. The personal computer or the like here refers to a computer having rich computing resources, and includes, for example, the learning device according to the embodiment described in the above first part. The state in which the source code operates on a personal computer or the like in this step 301 is the same state as step I described in the above "4-1".

[0153] In step 302, the source code written in Python or the like, which was confirmed to operate on a personal computer or the like in step 301, is caused to be executed by the CPU 201 of the embedded chip 200 using the evaluation board 100. Developers and the like confirm whether this source code can be operated by the CPU 201 based on the execution result. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Here, the source code written in Python or the like can be passed to the CPU 201 via the communication I / F 103 of the evaluation board 100 and the communication I / F 204 of the embedded chip 200. If it is found that this source code cannot be operated by the CPU 201, developers and the like correct this source code and repeat step 302. If it is confirmed that this source code can be operated by the CPU 201, developers and the like proceed to the next step 303.

[0154] In step 303, developers and the like rewrite the source code (at least a part thereof), which was confirmed to be operable by the CPU 201 in step 302, in a second programming language (for example, C language or assembler) in order to operate it on the accelerator 202.

[0155] In step 304, the source code rewritten in C language or the like in step 303 is executed on the accelerator 202 of the embedded chip 200 using the evaluation board 100. Developers and the like confirm whether the rewritten source code can operate on the accelerator 202 based on the execution result. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Here, the source code written in C language or the like can be passed to the accelerator 202 via the communication I / F 103 of the evaluation board 100 and the communication I / F 204 of the embedded chip 200. If it is found that this source code cannot operate on the accelerator 202, developers and the like modify this source code and repeat step 304. If it is confirmed that this source code can operate on the accelerator 202, developers and the like proceed to the next step 305.

[0156] In step 305, on the evaluation board 100, the result of the CPU 201 executing the first specific code (the code to be verified) in the source code written in Python or the like is compared with the result of the accelerator 202 executing the second specific code in the source code written in C language or the like, where the second specific code is the one obtained by rewriting the first specific code from Python or the like to C language or the like (for example, using a module called unit test executed by the embedded chip 200), and the comparison result is output. Developers and the like verify based on the comparison result whether the same output is obtained for the same input in both execution results. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Until this verification is completed, developers and the like repeat steps 303 to 305 described above. When this verification is completed, developers and the like proceed to the next step 306.

[0157] In step 306, the developer or the like tunes the source code written in C language or the like so that it can operate even faster by the accelerator 202 in step 305.

[0158] In step 307, on the evaluation board 100, the result of the CPU 201 executing the source code written in Python or the like is compared with the result of the accelerator 202 executing the source code written in C language or the like tuned in step 306 above (using, for example, a module called unit test executed by the embedded chip 200), and the comparison result is output. The developer or the like verifies based on the comparison result whether the same output is obtained for the same input in both execution results. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Until this verification is completed, the developer or the like repeats steps 306 and 307 described above. When this verification is completed, the developer or the like proceeds to the next step 308.

[0159] When step 307 is completed, the embedded chip 200 is in a state of operating according to two source codes, such as Python and C language. This state will be described with reference to FIG. 26. FIG. 26 is a schematic diagram showing the operating state of the embedded chip in the implementation method according to an embodiment of the present invention.

[0160] As shown in FIG. 26, in the above step 301 (which corresponds to step I), the calling side of the function (i.e., the entity that calls the function) is described in Python or the like, and the called side (i.e., the function to be called) is also described in Python or the like. Next, in the above steps 302 to 307, the calling side of the function is still described by Python or the like, and the called side is in a state where what is described by Python or the like and what is described by the C language or the like are mixed. That is, in the state where step 307 is completed, the embedded chip 200 is in a state of operating according to two source codes, such as Python and the C language.

[0161] What the implementation method according to this embodiment ultimately aims at is that, as shown at the right end of FIG. 26, both the calling side and the called side are described by the C language or the like, that is, the embedded chip 200 operates only according to the source code described by the C language or the like.

[0162] Therefore, returning to FIG. 25, in step 308, the developer or the like rewrites all the parts of the source code described in Python that have not yet been rewritten into the C language or the like so that the embedded chip 200 operates only according to the source code described in the C language or the like. In this step 308, the embedded chip 200 will be separated from Python. The source code described in the C language or the like generated in this way is stored in the external memory 205 or the like of the embedded chip 200. Thereby, the embedded chip 200 can read out the source code stored in the external memory 205 or the like and cause the accelerator 202 to execute it, and can execute machine learning. This state is the state aimed at by the implementation method according to the embodiment, and is the state in which the problems described in the above "1" and "2" are solved.

[0163] 5-2. Regarding the Second Method FIG. 27 is a schematic diagram showing a configuration example of a mounting apparatus used in a mounting method (second method) according to an embodiment of the present invention. The mounting apparatus (FIG. 27) used in the second method is different from the mounting apparatus (FIG. 24) used in the first method in that the embedded chip 200 does not include a CPU. In the second method, the operations that the CPU 201 of the embedded chip 200 performed in the first method are performed by a CPU provided in a computer (such as a personal computer) provided outside and not shown in the figure. For example, the computer (such as a personal computer) referred to here may be the learning apparatus (such as the personal computer illustrated in FIG. 11) described in the above first part.

[0164] In the mounting method performed by the second method, regarding the mounting method described with reference to FIG. 25, the operations executed by the CPU 201 of the embedded chip 200 in steps 302, 305, and 307 are changed to be performed by a CPU provided in the above-mentioned computer (not shown) provided outside. To achieve this, the evaluation board 100 shown in FIG. 27 may be communicably connected to the above-mentioned computer (not shown) provided outside via, for example, a communication I / F 103, so that the CPU provided in this computer executes the source code described in Python and the execution result can be received.

[0165] 6. Configuration of the mounting apparatus Next, the configuration necessary for the mounting apparatus 100 according to the above-described embodiment to implement the method described in the above "5" will be described.

[0166] 6-1. Definition of terms for explaining the configuration of the present invention (Difference between class and module) A module is a set of procedures and data defined and implemented to achieve a specific purpose (a concept independent of the support by a specific programming language). A class is a module defined and implemented using the support of an object-oriented language such as Python.

[0167] (Python layer and Native layer) The Native layer refers to the layer of the Native I / F and the implementation (software and hardware) called from there. The Python layer refers to the software layer that is assumed to be executed in the Python language. Currently, Chainer is written in the Python language, but it is also conceivable that Chainer will be ported to another programming language in the future. The functions described here as the Python layer do not necessarily mean that they are specialized for the Python language. As for the role sharing between the Python layer and the Native layer, the Python layer assumes a development environment with a higher abstraction level more suitable for algorithm design, and the Native layer assumes a development environment with a lower abstraction level that specifically takes into account the hardware configuration.

[0168] (Correspondence between computer and execution unit) FIG. 52 is a diagram showing an example of cooperation with an existing execution unit according to an embodiment of the present invention. The execution unit is a method of the Function / Optimizer class for actually calculating the algorithm of the neural network. The existing execution unit is a general-purpose computer execution unit or a GPU execution unit or both. The general-purpose computer execution unit calculates the algorithm of the NN using a general-purpose computer. The GPU execution unit calculates the algorithm of the NN using a GPU. The Native execution unit calculates the algorithm of the NN using the implementation of the Native layer. Since the Native layer is implemented for each type of computer, it can operate through the Native I / F for all types of computers (general-purpose computers, GPUs, accelerators).

[0169] 6-2. Configuration of the implementation unit FIG. 28 is a schematic diagram conceptually showing the functions of the mounting device according to an embodiment of the present invention. As shown in FIG. 28, the mounting unit 400 mainly includes a driving unit 401, a Function class / Optimizer class 402, a general-purpose computer execution unit 403, a GPU execution unit 404, a Native layer execution unit 405, a multi-dimensional array for general-purpose computer 406, a multi-dimensional array for GPU 407, a multi-dimensional array for Native 408, and a Variable class 409.

[0170] The driving unit 401 mainly includes an execution unit that instructs the Function class / Optimizer class 402 to execute a certain algorithm (function), and compares the execution result by the general-purpose computer execution unit 403 (or the execution result by the GPU execution unit 404) with the execution result by the Native layer execution unit 405 for the algorithm (function) using a module called, for example, a unit test, and outputs the comparison result.

[0171] The Function class / Optimizer class 402 causes at least one of the general-purpose computer execution unit 403, the GPU execution unit 404, and the Native layer execution unit 405 to execute the algorithm (function) instructed by the driving unit 401.

[0172] The general-purpose computer execution unit 403 acquires a multi-dimensional array corresponding to the algorithm (function) instructed by the Function class / Optimizer class 402 from the multi-dimensional array for general-purpose computer 406 and executes the algorithm (function) using a CPU. The execution result is returned to the driving unit 401 via the Function class / Optimizer class 402.

[0173] The GPU execution unit 404 acquires a multi-dimensional array corresponding to the algorithm (function) instructed by the Function class / Optimizer class 402 from the multi-dimensional array for GPU 407 and executes the algorithm (function) using a GPU. The execution result is returned to the driving unit 401 via the Function class / Optimizer class 402.

[0174] The Native layer execution unit 405 obtains a multi-dimensional array corresponding to the algorithm (function) commanded from the Function class / Optimizer class 402 from the Native multi-dimensional array 408 and executes the algorithm (function) using the accelerator. The execution result is returned to the drive unit 401 via the Function class / Optimizer class 402.

[0175] The Variable class 409 holds all the multi-dimensional arrays used by the general-purpose computer multi-dimensional array 406, the GPU multi-dimensional array 407, and the Native multi-dimensional array 408, and supplies the corresponding multi-dimensional arrays to the general-purpose computer multi-dimensional array 406, the GPU multi-dimensional array 407, and the Native multi-dimensional array 408.

[0176] Note that as an implementation method, when the first method described in the above "5-1" is adopted, all the components shown in FIG. 28 are arranged in the embedded system chip 200 (see FIG. 24). In this case, the general-purpose computer execution unit 403 executes the algorithm (function) using the CPU 201 mounted on the embedded system chip 200, the GPU execution unit 404 executes the algorithm (function) using the GPU (not shown) mounted on the embedded system chip 200, and the Native layer execution unit 405 mainly executes the algorithm (function) using the accelerator 202 mounted on the embedded system chip 200.

[0177] On the other hand, as an implementation method, when the second method described in the above "5-2" is adopted, among the components shown in FIG. 28, the Function class / Optimizer class 402, the general-purpose computer execution unit 403, the GPU execution unit 404, the multi-dimensional array 406 for the general-purpose computer, the multi-dimensional array 407 for the GPU, and the Variable class 409 are arranged in a computer (such as a personal computer) provided externally. In this case, the implementation of the Native layer is still arranged in the embedded chip 200. Further, in this case, the general-purpose computer execution unit 403 executes an algorithm (function) using the CPU of the externally provided computer, and the GPU execution unit 404 executes an algorithm (function) using the GPU of the externally provided computer.

[0178] 6-3.Configuration of the Native Layer Execution Unit Next, the configuration of the above-described Native layer execution unit 405 will be described. FIG. 29 is a schematic diagram showing a configuration example of the Native layer execution unit included in the implementation device according to an embodiment of the present invention. As shown in FIG. 29, the Native layer execution unit 405 mainly includes a NativeDevice class 501, a NativeArray class 502, a Function class / Optimizer class 503, and a bytecode generation unit 504 in the Python layer. Note that the Function class / Optimizer class 503 described in FIG. 29 and the Function class / Optimizer class 402 described in FIG. 28 are the same components, and the NativeArray class 502 described in FIG. 29 and the multi-dimensional array 408 for the Native described in FIG. 28 are the same components. Further, the Native layer execution unit 405 mainly includes a device management module 505, a data conversion module 506, a multi-dimensional array module 507, a Function module / Optimizer module 508, a virtual machine module 509, and a memory pool module 510 in the Native layer.

[0179] The NativeDevice class 502 wraps the device management module in the Native layer in the Python layer and hides function calls and data input / output to the Native layer. The NativeArray class 502 wraps the multi-dimensional array in the Native layer in the Python layer. Among the Function class / Optimizer class 503, the Function class wraps the Function module in the Native layer in the Python layer, and the Optimizer class wraps the Optimizer module in the Native layer in the Python layer. Here, the Function class and the Optimizer class are already implemented in Chainer and have the function of hiding the difference in execution between a general-purpose computer and a GPU. By extending this function, execution in the Native layer can also be hidden. The bytecode generation unit generates bytecode. Details of each component exemplified in FIG. 29 will be described later.

[0180] 7. Effects of the implementation device according to the embodiment Since deep learning is a technology in the stage of active research and development, it is assumed that a new layer algorithm with better performance than before will be invented during the development period for embedded chips, and there will be a need to incorporate such a new algorithm into the software or hardware implementation under development. In order to realize a state in which the configuration of a neural network including a new layer algorithm operates while satisfying the product-level specifications in an embedded environment, the following development steps need to be taken. 1. Implement and verify the algorithm in an environment where abundant computing resources such as a GPU can be obtained. 2. Combine the algorithm implemented and verified in 1 with the module of the neural network for which the optimization implementation has already been completed in the embedded chip and perform operation verification. According to the result, adapt the optimization specialized for the corresponding chip to the algorithm implemented and verified in 1. After the operation in 3.2 is completed, verify whether the product-level specifications are met after combining with other modules (such as sensor and motor control systems) using only the implementation of the neural network optimized for the corresponding chip through various test items.

[0181] When operating the neural network algorithm on Python, the implementation device according to the embodiment has a configuration that separately calls an execution unit using the Python language operating on a general-purpose computer, an execution unit using a GPU, and an execution unit using an optimized implementation for a specific chip for each layer, and has a configuration that operates the entire neural network algorithm using only this optimized implementation for the specific chip via bytecode. Between Steps 1 and 2 described in the previous paragraph, the algorithm implementation code created in Step 1 can be reused for Step 2, and the differences in the operation results between Steps 1 and 2 can also be easily compared and considered. Furthermore, between Steps 2 and 3, the optimized implementation created for Step 2 can be reused for Step 3, and conversely, the correction of defects related to the optimized implementation found in Step 3 can also be reused for Step 2. As a result, it is possible to realize, at the minimum development cost, a state in which a neural network configuration including a new layer algorithm operates while satisfying product-level specifications in an integration environment.

[0182] Definition of Terms The following terms are defined in the detailed description of the embodiments of the present invention. "Overall operation" refers to a processing unit that executes only forward processing or repeatedly executes forward processing, backward processing, and weight update processing. This overall operation is assumed as an embodiment of typical neural network learning and identification.

[0183] 8. Native Layer Next, the configuration related to the Native layer of the implementation device according to the embodiment illustrated in FIG. 29 will be described. 8-1. Device Management Module The device management module 505 performs initialization and release processing of devices (software and hardware states that depend on the optimization implementation). Specifically, the processing performed within the device management module 505 varies depending on the form of the device, but typical processing contents include, for example, securing and releasing a memory pool as described later. The device does not necessarily need to exist on the same chip or the same substrate as the general-purpose computer that executes Chainer and Python. It is also possible to implement an optimization that communicates with a device on a different substrate for initialization and release.

[0184] An example definition of a function for initializing or releasing a device is shown below. (Example 1) Device* chnr_init_device(void) This can initialize the device. (Example 2) void chnr_release_device(Device* device) This can release the device.

[0185] 8-2. Function module The Function module 508 is a group of functions that perform calculations for each layer of the neural network, and defines the following functions according to the type of layer. chnr_forward_xxxx(…) - Implement forward processing (floating-point version) chnr_backward_xxxx(…) - Implement backward processing (floating-point version) chnr_forward_xxxx_fx(…) - Implement forward processing (fixed-point version) chnr_backward_xxxx_fx(…) - Implement backward processing (fixed-point version) Here, xxxx represents a name assigned according to the type of layer. The specific processing contents in each function include those exemplified in the above "2-6" to "2-12" of the above first part.

[0186] 8-3. Multidimensional Array (MDArray) Module The multidimensional array module 507 manages the multidimensional arrays that are input and output between the functions in the Native layer. The multidimensional array module 507 can manage arrays of any size and number of dimensions. Also, as will be described later, the multidimensional array module 507 has a mechanism for mutual conversion with Numpy (a multidimensional array class in the Python layer on which Chainer depends) and a multidimensional array library for GPUs. Furthermore, this multidimensional array module 507 can hold not only floating-point types but also fixed-point types. As a result, neural network calculations can be easily realized even on hardware that does not have an FPU (floating-point arithmetic unit). Also, this multidimensional array module 507 has a function for mutual conversion with floating-point multidimensional arrays.

[0187] An implementation example of the multidimensional array module 507 will be described. An example of the structure definition of the multidimensional array module of the implementation device according to an embodiment of the present invention is shown in FIG. 30. Next, an example of the function definition is as follows. (Example 1) MDArray chnr_create_md_array(dimensions[], numaxis, type) This makes it possible to generate and initialize a multidimensional array. (Example 2) void chnr_delete_md_array(MDArray* mdrray) This makes it possible to delete a multidimensional array. (Example 3) void chnr_md_array_add(MDArray* dst, MDArray* a, MDArray* b) This makes it possible to add the elements of multidimensional arrays to each other.

[0188] Next, the memory management of the multidimensional arrays of the multidimensional array module 507 will be described. The management (generation / destruction) of the memory area that stores the entity of the multi-dimensional array is realized in the Native layer. In the case of an embedded environment, there may be an environment with a memory configuration that cannot be managed by the memory management mechanism (malloc / free) provided as a standard by Linux (registered trademark) OS or the like. Considering the role sharing of software layers, such as algorithm development in the Python layer and development with a strong awareness of hardware in the Native layer, it is appropriate to implement the management mechanism responsible for the characteristics of such a hardware environment in the Native layer. Even when using the virtual machine described later (when the dependence on the Python layer is removed), this memory management mechanism can be reused. Prepare a class in the Python layer that wraps the multi-dimensional array in the Native layer, and match the timing of memory area generation / release with the instance lifetime of this Python class. Such a mechanism is necessary to handle multi-dimensional arrays naturally in the code on Python. The "Define-by-Run" function also depends on the memory management mechanism of Python.

[0189] The mutual compatibility and reference relationships of multi-dimensional array data are shown in Fig. 31.

[0190] 8-4. Memory Pool Module The memory pool module 510 is a mechanism for reducing the number of calls to a memory management mechanism with high cost (such as the number of processing cycles) by reusing the memory area once secured. An example of a function definition is as follows. (Example 1) void chnr_momory_pool_init(MemoryPool* momory_pool) This enables the initialization of the memory pool. (Example 2) void chnr_momory_pool_release(MemoryPool* momory_pool) This enables the destruction of the memory pool. (Example 3) void* chnr_momory_pool_alloc_buffer(MemoryPool* momory_pool, int byte_size, void* old_addr) This enables the allocation of memory. (Example 4) void chnr_momory_pool_free_buffer(MemoryPool* momory_pool, void*addr) This enables the release of memory.

[0191] Background for the need of a memory pool module in the Native layer (1) Chainer's "Define-by-Run" relies on the dynamic memory management mechanism of Python. An example (forward processing of the Linear Function) is shown in Fig. 32. In this example, an instance of Wx (the content is a multi-dimensional array) is newly generated in the description of Wx = x.dot(self.W.T) on the third line. When Wx is no longer referenced by any variable, it is automatically discarded by Python's memory management mechanism. The size of the data output by the Function (Wx in the above example) can be dynamically changed depending on the input data size and parameters, and the allocation of its entity (memory area) is also executed within the code flow of forward processing and backward processing. In order to realize "Define-by-Run" (defined at the time of network configuration execution), a mechanism for allocating only the necessary size of memory at such necessary timings is required.

[0192] Background for the need of a memory pool module in the Native layer (2) By preparing a class in the Python layer to wrap the multi-dimensional array in the Native layer and devising to match the lifetime of the multi-dimensional array in the Native layer with that in the Python layer, the implementation in the Native layer can be utilized while enjoying the flexibility of "Define-by-Run". However, since the Function is usually called frequently, if the high-cost memory management mechanisms such as malloc and free in the Native layer are called each time, it may lead to a decrease in processing speed. Therefore, it is necessary to prepare a function (memory pool) to reuse the once-allocated memory area.

[0193] Implementation example of memory pool (1) An example of a structure definition is shown in Figure 33. The processing flow when allocating memory is as follows. 1. Search for an index in the Buffer_size array where the freed flag is 1 and the size at the previous allocation matches the size to be allocated this time. If found, set the freed flag to 0 and return the value of buffer_addr at the same index (the address of the memory buffer). Here, the freed flag is managed by, for example, the sign bit of the buffer_size array element. By searching the array elements based on the combination of the size and address at the previous allocation, the address swapping can be reduced. 2. If the above-matching index is not found, actually allocate memory (call malloc, etc.), add the address and size to the array, and then return the address.

[0194] Implementation example of memory pool (2) As the processing when freeing memory, search for the address to be freed in the buffer_addr array. If the address is found, set the freed flag to 1. As the processing when freeing the memory pool, free the memory (call the free function, etc.) for the elements in the buffer_addr array where the address is set.

[0195] Effect of memory pool implementation In most neural networks, since the combination of memory sizes is fixed and does not change for each iteration of learning, by using the implementation example of the memory pool as described above, calls such as malloc can be limited to only the first iteration.

[0196] 8-6. Optimizer Module Optimizer module 508 is a group of functions that perform weight updates for each layer having the weights of the neural network. Optimizer module 508 defines the following functions according to the weight update algorithm. (Example 1) chnr_op_init_state_xxxx(…) This enables the implementation of the internal state initialization process of the weight update algorithm (floating-point version). (Example 2) chnr_op_update_one_xxxx(…) This enables the implementation of the weight update process (floating-point version). (Example 3) chnr_op_init_state_xxxx_fx (…) This enables the implementation of the internal state initialization process of the weight update algorithm (fixed-point version). (Example 4) chnr_op_update_one_xxxx_fx (…) This enables the implementation of the weight update process (fixed-point version). Here, xxxx represents the name given according to the weight update algorithm. Note that the weight update algorithm can include those described in the above "2-13" of the above first part.

[0197] 8-7. Data Conversion (Converter) Module (1) The data conversion module 506 is a group of functions that perform data format conversion. Examples of function definitions are as follows. (Example 1) chnr_float_to_fixed(MDArray* dst, MDArray* src, int Q) This enables conversion from floating-point type to fixed-point type. (Example 2) chnr_fixed_to_float(MDArray* dst, MDArray* src) This enables conversion from fixed-point type to floating-point type. (Example 3) chnr_host_to_device(MDArray* dst, float* src_data, int src_dimensions[], int src_num_axis, int Q, int async, …) This enables conversion from device-independent data representation (described later) to device-dependent data representation (described later). (Example 4) chnr_device_to_host(float* dst_data, int dst_dimensions[], int*dst_num_axis, MDArray* src, int async, …) This enables conversion from device-dependent data representation to device-independent data representation.

[0198] Effect of floating-point type and fixed-point type conversion For embedded semiconductor chips, many aim to reduce hardware resources (number of transistors and power consumption) relative to the amount of computation by omitting the FPU (floating-point unit) or adopting a circuit design that does not use the FPU, at least for large-scale parallel computing. When executing numerical algorithms such as neural networks without using an FPU, a data type called fixed-point type is often used, which uses an integer arithmetic unit and a shift arithmetic unit to represent numerical values including information below the decimal point. The floating-point type is a data type suitable for algorithm design in the sense that it allows more intuitive handling of real values, while the fixed-point type can be said to be a data type suitable for effective utilization of hardware resources. By preparing a conversion function between such data types within the framework for designing and executing a neural network, the development of neural network algorithms can be gradually advanced from a mathematical perspective to an implementation perspective considering hardware in a unified environment, while checking the degree of influence of type conversion unit by Function.

[0199] Device-independent data representation refers to a data representation that does not have information dependent on a specific computer. A typical implementation of this data representation is a multi-dimensional array in the C language format with consecutive memory addresses. Such a data representation can be easily handled by using libraries such as numpy at the Python layer, but it is not specific to a library or a host language. Device-dependent data representation refers to a data representation suitable for an optimized implementation specialized for a specific computer. By preparing a function to mutually convert these two data representations, the overall operation can be executed in cooperation with an optimization implementation strongly considering hardware and an implementation considering algorithms (such as readable code with a structure similar to mathematical formulas described in Python).

[0200] Examples of requirements to consider in conversion to device-dependent data representation (1) Memory configuration Should it be placed in shared memory? Should it be placed in a hardware-specific memory area? (2) Memory alignment Start address, start address for each dimension, padding (3) Byte order Little endian, big endian (4) Data type Fixed-point number (Q value) / floating-point number, byte width (32bit, 16bit, 8bit, …) (5) Scheduling of data input / output in multi-core execution (6) Data communication Communication processing when a device with an optimized implementation is on a separate chip, separate board, etc.

[0201] 8-8. Communication means By applying the following changes to the implementation of the function group (Native I / F) in the Native layer described so far, the overall operation can be executed at high speed while communicating with a device on a separate chip, separate board, etc. (1) RPC (remote procedure call) (2) Instruction queue (3) Reduction of data communication volume for multi-dimensional arrays (4) Asynchronous processing of transfer and calculation The following terms are defined to explain these change policies "Host device": The device that executes the overall operation (the device on which the Chainer code is executed on Python in the normal implementation) "Remote device": A device that requires communication processing for reasons such as being on a separate chip or separate board

[0202] RPC (remote procedure call) When a function defined in the Native I / F is called, instead of directly executing the processing requirements (memory allocation and operation execution), information representing the processing request (function type and arguments) is generated and sent to the remote device. The remote device executes the processing based on the instruction, and then the host device receives the processing result. Instruction queue Rather than executing the communication of processing requests by RPC each time a function defined in the Native I / F is called, the communication schedule is optimized by temporarily storing the information representing the processing requests in a queue (FIFO buffer). Reduction of data communication volume of multi-dimensional arrays Since multi-dimensional arrays have a huge data size, reducing their communication volume is an important issue in improving the overall operation speed. There are roughly two major strategies for reducing the communication volume. (1) Reduce the number of transfers of multi-dimensional arrays (2) Reduce the data communication volume of individual multi-dimensional arrays

[0203] Method for reducing the number of transfers of multi-dimensional arrays The data and weight gradients input and output between layers of the intermediate layer (other than the input and output layers of the neural network) only need to exist on the remote device, and communication between devices is unnecessary. Also, for the "weights", they only need to be transferred to the remote device at the first stage of defining the network structure and transferred to the host device at the end of learning. The conversion functions for device-independent data representation and device-dependent data representation described for the data conversion (Converter) module 506 are optimal for managing such transfer timings. Specifically, each function performs the following processing. When converting from device-independent data representation to device-dependent data representation, data transfer is performed from the host device to the remote device. When converting from device-dependent data representation to device-independent data representation, data transfer is performed from the remote device to the host device.

[0204] Method for reducing the data communication volume of individual multi-dimensional arrays Various algorithms for data compression are known. (1) Reversible compression (Huffman coding, run-length compression, etc.) (2) Irreversible compression (DCT, scalar quantization, vector quantization, etc.) By specifying the types and parameters of these compression algorithms as arguments in a function that requests data communication (assuming a conversion function between device-independent data representation and device-dependent data representation), the communication volume can be reduced using an optimal data compression method based on the nature of the data and accuracy requirements.

[0205] Asynchronous processing of transfer and calculation In many embedded chips, the configuration is such that separate hardware executes data communication and arithmetic processing asynchronously. If a function that requests data communication (assuming a conversion function between device-independent data representation and device-dependent data representation) is prepared to be non-blocking, the overall speed of the algorithm can be increased by coding while being aware of such a hardware configuration (a technique generally called pipelining). Figure 34 shows the coding of pipelining in pseudocode

[0206] 8-9. Virtual machine module The virtual machine module 509 is a group of functions that realizes the function of interpreting bytecode and executing neural network learning / identification processing (forward processing, backward processing, and weight update). The bytecode is assumed to be generated by a bytecode outputter in the Python layer described later, but even bytecode generated by other software can be interpreted and executed by the virtual machine module if the format is correct. An example of a function definition is as follows. (Example 1) void chnr_init_with_bytecode(VMState* state, char* byte_code) This enables syntax analysis of the bytecode and initialization of the internal state of the virtual machine. (Example 2) void chnr_forward(VMState* state) This enables execution of forward processing. (Example 3) void chnr_backward(VMState* state) As a result, backward processing can be executed. (Example 4) void chnr_update_weight(VMState* state) As a result, weight update processing can be executed.

[0207] Bytecode format example Store the following information in binary format. (1) Input / output data information {Array dimension number and size, data type (float32, FixedPoint)} * Number of Variables (2) Weight information {Array dimension number and size, data type (float32, FixedPoint), actual value} * Number of weights (3) Function call information during backward processing {Function type, index of input / output data, index of weight information, unique parameters for each function type} * Number of functions (4) Type and parameters of weight update Furthermore, the bytecode may include the index of the multi-dimensional array that serves as the input and output of the entire neural network processing. By storing this index in the bytecode, user code that uses the virtual machine can appropriately associate the multi-dimensional array that serves as the input of the entire neural network processing with the multi-dimensional array that serves as the output with respect to function calls. For example, this association can be performed according to the following flow. (Step 1) The user code obtains the input multi-dimensional array of the entire processing by calling a function prepared in the configuration of the virtual machine. (Step 2) The user code copies the input data to the multi-dimensional array obtained in Step 1. (Step 3) The user code calls a function that executes the entire operation prepared in the configuration of the virtual machine. (Step 4) The user code obtains the output multi-dimensional array of the overall process by calling a function prepared for the configuration of the virtual machine (this multi-dimensional array is in a state where the processing results of the overall operation executed in the above step 3 are stored. The functions in step 3 and step 4 do not necessarily have to be separated and may be an integrated function.). (Step 5) The user code obtains the content of the output data from the multi-dimensional array obtained in the above step 4.

[0208] Implementation example of the processing flow for initializing the internal state of the virtual machine (1) Interpret the "input / output data information" in the bytecode to generate a list of multi-dimensional arrays for input / output by the Function. (2) Interpret the "weight information" in the bytecode to generate a list of weights and weight gradients (both are multi-dimensional arrays). (3) Interpret the "Function call information during Backward" in the bytecode to generate a list of structures (FunctionState) with the following information for Forward and Backward respectively (the identification ID of the function for executing the forward process / backward process, the address of the input / output data, the address of the weight, the address of the weight gradient, and parameters specific to each type of Function). (4) Interpret the "type and parameters of weight update" in the bytecode to initialize the multi-dimensional array of the internal state of the weight update algorithm and a structure (OptimizerState) with the following information (the address of the function for executing the weight update, the address of the weight, the address of the weight gradient, the internal state of the weight update algorithm, and parameters specific to each type of weight update).

[0209] The configuration diagram of the internal state of the virtual machine is shown in Fig. 35

[0210] Execution flow example of the virtual machine module (1) (Forward process and backward process) The virtual machine module executes processing like the pseudo-code shown in Fig. 36.

[0211] Execution Flow Example of Virtual Machine (2) (Optimizer) The virtual machine module executes processing such as the pseudo-code shown in FIG. 37.

[0212] Configuration 2 of the optimization device specialized for executing Forward processing; when having means for reusing the data memory area (1) When executing the overall operation, if only the identification process is executed without performing learning (weight update), only the forward process needs to be executed. In this case, the following data becomes unnecessary. (1) Among the data input and output between layers, those not accessed by the currently executing Function (2) Gradient of weights (3) Internal state of the weight update algorithm When initializing the internal state of the virtual machine, there is no need to secure the gradient of the weights and the internal state of the weight update algorithm. Regarding the data input and output between layers, the amount of memory to be secured can also be suppressed by procedures such as those described in the next paragraph, for example.

[0213] Configuration 2 of the optimization device specialized for executing Forward processing; when having means for reusing the data memory area (2) Example of procedure at the time of initializing the internal state of the virtual machine module (1) Calculate the sum of the data sizes (memory sizes) input and output for each Function, and select the one with the maximum size. (2) When initializing the structure (MDArray) for handling multi-dimensional arrays, set the address so as to reuse the memory area secured in step 1. By performing address setting such that the left and right ends of the memory area are alternately switched as input and output for each layer and used, it is possible to prevent the copying of array data. When a Function that performs input and output with a loop is included, for the output data carried over to the next iteration, a memory area is secured separately as not being subject to the reuse shown in this procedure.

[0214] Configuration 2 of the optimization device specialized for Forward process execution; when having a means for reusing the data memory area (3) An example of the address setting of the data input and output by the Function is shown in FIG. 38.

[0215] Supplementary explanation regarding the format of the bytecode In the previous explanations, examples of simply implementing the information stored in the "Function call information during backward processing" in ascending or descending order were shown. However, by storing multiple execution orders in the bytecode or storing instructions for repetition and branching in the bytecode, it is also possible to execute more advanced processing such as dynamically changing the configuration of the neural network according to the nature of the input data during virtual machine execution. The memory management mechanism described above for the memory pool module can be used to realize such a dynamic mechanism.

[0216] Regarding the labeling of the data input and output from the virtual machine to external code A list of the data input and output between Functions is created in the "Initialization process of the internal state of the virtual machine". However, the simplest method is for the external code that calls the functions of the virtual machine to directly access the elements of this list. If the variable name of the Python Variable instance is stored in the "Input / output data information" during bytecode generation, the input and output can also be labeled using this name.

[0217] 9. Python layer Next, the configuration regarding the Python layer of the implementation device according to the embodiment illustrated in FIG. 29 will be described. 9-1. NativeArray class The NativeArray class 502 is a class that wraps the multi-dimensional arrays in the native layer at the Python layer. The NativeArray class 502 is generated as an instance that corresponds one-to-one with the multi-dimensional arrays in the native layer. Also, the NativeArray class 502 has a lifetime management function based on reference counting as a basic function as a Python object. Furthermore, the NativeArray class 502 has a function to request the release of the multi-dimensional arrays in the native layer when the lifetime ends. Also, the NativeArray class 502 has a function to hold a copy of the type information of the multi-dimensional arrays in the native layer and pass it on to other objects in the Python layer. Furthermore, the NativeArray class 502 has functions such as data copying and addition for each array element, and has a function to request the execution of these in the native layer. In addition, the NativeArray class 502 has functions to operate compatibly with multi-dimensional array libraries in the Python layer such as Numpy and GPUArray on which Chainer depends.

[0218] 9-2. NativeDevice Class The NativeDevice class 501 is a class that abstracts the optimized implementation and reference implementation in the native layer. The NativeDevice class 501 has a function to request the following processing to the native layer in response to requests from other objects in the Python layer. (1) Initialization and release of the device (2) Generation and copying of multi-dimensional arrays (generate a NativeArray instance in the Python layer that wraps this) (3) Conversion between device-independent data representation and device-dependent data representation (conversion between floating point and fixed point can also be instructed) (4) Execution of processing of Function and Optimizer (dispatch individual functions in the native layer)

[0219] 9-3. Function Class The Function class 503 is a class that defines forward processing and backward processing in pairs. The Function class 503 exists in Chainer, but it adds the function of requesting forward processing and backward processing to the Native layer. An example of method implementation is as follows. (Example 1) forward_native(…) This enables the request for forward processing to the Native layer. (Example 2) backward_native(…) This enables the request for backward processing to the Native layer.

[0220] The processing flow assumed when calling Forward_native or backward_native (1) Calculate the output data size from the input data size and the parameters at the time of Function instance initialization. (2) Pass the output data size obtained in (1), the instance of the input data (NativeArray), and the classification of the Function (Linear, ReLU, …) to the NativeDevice instance to request a function call to the Native layer. (3) The NativeDevice instance executes the following processing in response to this call. (A) Request the Native layer to generate a multi-dimensional array of output data. This processing is not performed for a Function that overwrites the input data. (B) Determine and call the function of the Native layer actually to be called based on the type of the input data (floating point, fixed point) and the classification of the Function (the function of the Native layer writes the processing result to the multi-dimensional array secured in A) (C) Generate a NativeArray instance that wraps the multi-dimensional array secured in (A) above. (4) Return the NativeArray instance generated in (C) above as the return value of the Function.

[0221] 9-4. Optimizer Class The Optimizer class 503 is a class that performs weight updates. The Optimizer class 503 is a class existing in Chainer, but it adds a function to request state initialization and weight update processing to the Native layer. An example of method implementation is as follows. (Example 1) init_state_native(…) This enables the request of the internal state initialization process of the weight update algorithm to the Native layer. (Example 2) update_one_native(…) This enables the request of the weight update process to the Native layer. The processing flow at the time of these method calls is equivalent to that already explained in the above "Function class".

[0222] 9-5. Specific Example of the Overall Operation in Cooperation with the Native Layer A specific example is illustrated in Fig. 39.

[0223] 9-6. Bytecode Generation (Output) Section The bytecode generator 504 is a mechanism that converts the network configuration of the neural network defined by "Define-by-Run" into bytecode (an interpretable and executable data format) and outputs it. As for the format of the bytecode, for example, those described above regarding the "virtual machine module" can be considered. However, in addition to such a format, outputs to the following formats can also be considered, for example. (1) Neural Network Definition Format of Caffe It can be executed in Caffe (Caffe is one of the representative frameworks for neural network design and execution). (2) Programming Languages such as C and Java (registered trademark) Software that executes the overall operation can be generated. (3) Hardware Description Languages such as HDL and Verilog Hardware that executes the overall operation can be synthesized.

[0224] An example of the function definition of the bytecode generation unit is as follows. Function name: write_network_difinition(output_node, path, format) Function specification: Output the network configuration that connects from the output_node towards the input side to the file specified by path in the format specified by format. output_node can be specified as a list (multiple nodes can be used as the starting point).

[0225] An example of the procedure for outputting bytecode from the reference structure for backward processing As described in the above first part, Chainer has a function to generate a reference structure for backward processing according to the description of the natural forward processing calculation. Since the forward processing can be calculated by tracing the reference structure for backward processing in reverse order, if bytecode is generated from this reference structure, both the forward processing and the backward processing can be executed. This procedure can be roughly divided into the following steps. (1) Generation of element information for bytecode creation Generation of input / output data information Generation of weight information Generation of Function call information during backward processing (2) Conversion of element information into bytecode

[0226] Procedure for generating element information for bytecode creation - Traverse the "reference structure for backward processing" starting from the output_node passed to write_network_difinition and execute the following processing. (1) If the current node is a Variable, add the information of its multi-dimensional array (size, number of dimensions, floating point / fixed point (Q value)) to the list of "input / output data information". (2) If the current node is a Function, perform the following processing. (i) Add the information of the multi-dimensional array of weights (size, number of dimensions, floating point / fixed point (Q value), realized value of the weights) to the list of "weight information" without allowing duplicates (since multiple Function instances can share the same weights). (ii) Add the type of Function, the index of the input / output data, the index of the weights, and the parameters specific to each type of Function to the list of "Function call information during backward processing". When multiple starting nodes are passed to the output_node, avoid duplicate registration of the same node in the procedure of the next paragraph.

[0227] Procedure for creating element information when multiple starting nodes are passed (1) Create an empty list of "Function call information during backward processing". (2) For each starting node in the output_node, perform the following procedure. (A) Create a list of "Function call information during backward processing" specific to the starting node. (B) Perform the registration procedure described in the previous paragraph on the list created in (A) above. At this time, avoid duplicate registration by not performing the registration procedure for nodes that are already registered in the list created in (1) above. (C) Concatenate the list created in (A) above to the front of the list created in (1) above.

[0228] Python layer: Bytecode outputter (6) Conversion of element information to bytecode Convert the following information created in the procedure of "Generating element information for bytecode creation" to the format specified by the format argument of write_network_difinition. (1) Generation of input / output data information (2) Generation of weight information (3) Generation of Function call information during backward processing Examples of formats include those exemplified earlier for the "bytecode generation section".

[0229] Output of multiple network configurations The write_network_difinition function described above for the "bytecode generation section" is designed to directly write the network configuration to the file passed as the argument path. However, it is also possible to pass an object for writing the configurations of multiple networks to the bytecode to this path argument. Here, the network configuration refers to the components (1), (2), and (3) explained in "Python layer: bytecode outputter for element information (6) conversion of element information to bytecode". This "object for writing the configurations of multiple networks to the bytecode" shares the same "(2) weight information" among the "multiple network configurations" and reduces the weight information to be written to the bytecode. For "(1) input / output data information generation" and "(3) function call information generation during backward processing", even if there are partial information overlaps, they are generated independently according to the above steps. A code example when using this object is shown in Figure 40. In this code example, on the 6th line, the network configuration is output by tracing the reference structure for backward processing from nodeA, and on the 7th line, the network configuration is output by tracing the reference structure for backward processing from nodeB. On the 8th line, these two network configurations are output to one file (. / bytecode.bin).

[0230] Method for specifying different function call orders for forward and backward processing of the same network As described in the above first part, Chainer has a function (unchain_backward method) to cut off the reference structure for Backward from a specific Variable instance towards the input layer side. By combining this unchain_backward method with the "output of multiple network configurations" explained in the previous paragraph, it is also possible to specify different function call orders for the calculations of forward and backward processing in the same network. In the code example shown in FIG. 41, a network definition that executes all processes from A to D is output in the call of #1, while a network definition that executes only the processes from B to D is output in the call of #2. When executing bytecode by a virtual machine, it is possible to make different uses such as performing Forward processing on the network configuration output by #1 and performing Backward processing on that of #2.

[0231] 10. Configuration common to the Native layer and the Python layer 10-1. Algorithm execution unit that fuses multiple NN algorithms Combinations of frequently appearing Functions can be seen in the configuration of general neural networks. (Example 1) Linear → ReLU, Linear → ReLU → Linear → ReLU (Example 2) Convolution2D → ReLU, Convolution2D → ReLU → Convolution2D → ReLU By defining such frequently appearing combinations of Functions as one Function and implementing a specialization for that calculation in both the Python layer and the Native layer, the following advantages can be enjoyed in both algorithm design and hardware execution efficiency. (1) The overhead (function calls and communications) for calling Functions can be reduced. (2) High execution efficiency can be obtained by implementing while considering data dependency relationships and parallelism spanning multiple Functions (such as effectively utilizing cache memory and reducing the amount of data directly accessing the main memory). (3) By using more abstract Functions during algorithm design, it becomes easier to understand and define complex network configurations.

[0232] Generally, in recent years, while the speed of computing cores in computers has increased significantly, the performance of memory access has not been increased as much. For this reason, when looking at the performance of the entire computer, there is a problem that the performance is limited by memory access and sufficient computing performance cannot be achieved. To solve such a problem, a mechanism is used to place a small but particularly fast memory called a cache memory and a register file close to the computing core physically, and perform many calculations on the cache memory to bridge the speed gap between the two.

[0233] By the way, the following combinations of Functions that frequently appear can be seen in the configuration of neural networks. ·Convolution2D→ReLU ·Convolution2D→ReLU→Convolution2D→ReLU Since Convolution2D has a large amount of computation with respect to the data size of input and output, there is a high chance to effectively utilize mechanisms such as cache memory and demonstrate the performance of the computing core. On the other hand, since ReLU has a small amount of computation with respect to the data size of input and output, this chance is small. When Convolution2D and ReLU are executed as individual Functions, after writing out all the data of the processing result of Convolution2D to the main memory, it is necessary to transfer the content back to the periphery of the computing core and perform the calculation of ReLU. The reason is that it is not known immediately whether the result will be used by ReLU after the processing of Convolution2D is completed.

[0234] Therefore, if Convolution2D and ReLU are executed as an integrated Function, before writing out the processing result of Convolution2D to the main memory, it can be directly used as the input for the processing of ReLU on the cache memory or the register file. Thus, the frequency of data transfer to the main memory is reduced, and the chance to execute the processing more efficiently (faster) increases. If more functions can be executed as an integrated function, such as Convolution2D→ReLU→Convolution2D→ReLU, the chance of improving processing efficiency will further increase. This is because by considering the size of the cache memory and the data dependency within the combination of functions, the amount of access to the main memory can be more actively reduced.

[0235] 10-2.Configuration of the optimization device specialized for Forward processing execution 1; when having the weight optimization processing means Among the layer algorithms of neural networks, there are some that can reduce the amount of computation and memory usage by performing weight optimization specialized for the case of only executing Forward processing without performing Backward processing. The reasons for such optimization are as follows. To train a neural network using stochastic gradient descent, a high numerical accuracy and a high degree of freedom in the value range are required for the weight vector. This is because it is necessary to accumulate small value updates during learning, and it is not possible to sufficiently assume the value range in which the weight vector will change in advance. On the other hand, if only Forward processing is executed, such a trade-off between accuracy and freedom is not necessary. By reducing the amount of weight information and then executing Forward processing, the memory and computation can be reduced. The computation can be reduced because measures such as reducing the number of weight elements and not calculating zero weights can be taken. For example, in the processing of Linear (inner product between layers), there is a technique known to compress the data size of the weights and reduce the calculation size by performing singular value decomposition on the weight information (a matrix of the number of input nodes * the number of output nodes) and deleting elements with small diagonal components. (J. Xue, J. Li, and Y. Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, 2013) By adding a method to the Function class in Chainer that executes such weight optimization processing specialized for Forward, it is possible to reduce the computational resources when executing only the Forward process using the already learned weights. This method, like the existing Forward method and Backward method in the Function class, has a function of hiding the differences in hardware implementations (general-purpose computer, GPU, Native) (calling different re-initialization implementations) depending on the type of the multi-dimensional array of weights held by the Function.

[0236] Supplementary Explanation on Software-Hardware Configuration So far, the details of the embodiments have been described by exemplifying that specific functions are subordinate to functions and classes in the Python layer and the Native layer. However, the role sharing of these software layers, classes, and functions is merely an example for specifically explaining the functional configuration according to the embodiments of the present invention, and as shown in the following examples, it is also conceivable that individual functions according to the embodiments of the present invention are implemented in classes, layers, or hardware different from the previous explanations. (1) The processing content described in "Configuration 2 of the Optimization Device Specialized for Executing Forward Processing; When Having a Means for Reusing the Data Memory Area" can be executed in advance by the bytecode outputter instead of the virtual machine module. (2) The Function described in "Function Combining Multiple Functions" can be implemented by specialized hardware (FPGA, ASIC) instead of software-level optimization. Therefore, the configuration according to the embodiments of the present invention does not directly depend on functions and classes in the Python layer and the Native layer, or an implementation assuming software.

Explanation of Signs

[0237] 10 Learning Device 100 Evaluation Board 110 Acquisition Unit 120 Storage Unit 130 Execution Unit 200 Embedded Chips (Embedded Semiconductor Integrated Circuits) 401 Drive Unit 402 Function Class / Optimizer Class 405 Native Layer Execution Unit 408 Multidimensional Array for Native 409 Variable Class 504 Bytecode Generation Unit 505 Device Management Module 506 Data Conversion Module 507 Multidimensional Array Module 508 Function Module / Optimizer Module 509 Virtual Machine Module 510 Memory Pool Module

Claims

1. A data generation method for executing, in a semiconductor integrated circuit, an operation according to a neural network trained using a deep learning framework, comprising: At least one processor optimizing weights of the neural network trained using the deep learning framework after the training; generating data for executing the calculation on the semiconductor integrated circuit, the data including the optimized weights, based on the neural network; the optimizing includes compressing a data size of the neural network weights by the at least one processor; the data is in a data format that can be interpreted and executed by the semiconductor integrated circuit, and the calculation is performed using the optimized weights by interpretation and execution by the semiconductor integrated circuit; The operation includes forward processing of the neural network. Data generation method.

2. the optimizing includes the at least one processor reducing the number of weight elements of the neural network. The data generation method according to claim 1 .

3. the at least one processor reduces the number of elements based on singular values ​​of a matrix representing the neural network weights; The data generation method according to claim 2 .

4. the at least one processor generates the data including weights of other neural networks; The data generating method according to any one of claims 1 to 3.

5. the at least one processor generates the data including weights shared between the neural network and the other neural network; The data generation method according to claim 4 .

6. the optimized weights include weights expressed in fixed point; The data generating method according to any one of claims 1 to 5.

7. the at least one processor generating the data including information about invocations of functions; The data generating method according to any one of claims 1 to 6.

8. the at least one processor generates the data including, as the information regarding the function invocation, information regarding the invocation of a third function that combines functions of a first function and a second function used in the deep learning framework; The data generation method according to claim 7 .

9. the first function and the second function are functions related to a layer algorithm; The data generation method according to claim 8.

10. the first function is either a matrix multiplication function or a convolution function; the second function is either an activation function or a normalization function. The data generation method according to claim 8.

11. the at least one processor obtains information about the function invocation based on a reference structure generated by training the neural network using the deep learning framework; The data generating method according to any one of claims 7 to 10.

12. The reference structure is a data structure including information on the execution order of forward processing or backward processing of the neural network. The data generation method according to claim 11.

13. the at least one processor generates the data including indices of multidimensional arrays that are inputs and outputs to the neural network; The data generating method according to any one of claims 1 to 12.

14. The neural network is trained using a programming language higher than C. The data generating method according to any one of claims 1 to 13.

15. the at least one processor generates the data including information regarding a network configuration of the neural network. The data generating method according to any one of claims 1 to 14.

16. the at least one processor generates the data including information about network configurations of a plurality of neural networks including the neural network; The data generating method according to any one of claims 1 to 15.

17. the at least one processor generates the data without including information used only for performing backward processing.

17. The data generation method according to any one of claims 1 to 16.

18. The information used only for performing the backward processing includes at least the gradient of the weights and the internal state of the weight update algorithm.

18. The data generation method according to claim 17.

19. the at least one processor generates address information for data input / output between layers of the neural network, the address information being used to reuse a memory area of ​​the semiconductor integrated circuit; 19. The data generation method according to any one of claims 1 to 18.

20. the at least one processor generating the data including information regarding the execution order of functions; 20. The data generation method according to any one of claims 1 to 19.

21. The semiconductor integrated circuit comprises: Memory and a control processor; an accelerator; 21. The data generation method according to any one of claims 1 to 20.

22. the at least one processor: training the neural network using the deep learning framework that supports the Define-by-Run function; 22. The data generation method according to any one of claims 1 to 21.

23. The at least one processor performs the learning process by: Execute forward processing of the neural network; By executing the forward processing, information regarding a calculation procedure for the backward processing is generated; performing the backward processing of the neural network based on the generated information about the calculation procedure; updating the parameters of the neural network based on the execution result of the backward processing; 23. The data generation method of claim 22.

24. the at least one processor generates a data structure as information relating to the calculation procedure; 24. The data generation method of claim 23.

25. The data structure is not generated before the forwarding process is performed.

25. The data generation method of claim 24.

26. The at least one processor performs the learning process by: In addition to executing the forward processing of the neural network, generate information related to the calculation procedure.

26. The data generation method according to any one of claims 23 to 25.

27. The at least one processor performs the learning process by: Executing the backward process in the reverse order of the forward process based on the generated information on the calculation procedure.

27. The data generation method according to any one of claims 23 to 26.

28. The at least one processor performs the learning process by: performing the forward processing using a first object; switching whether or not to generate information on the calculation procedure for the first object based on an attribute of the first object; 28. The data generation method according to any one of claims 23 to 27.

29. The at least one processor performs the learning process by: if the attribute of the first object is valid, generating information about the calculation procedure for the first object; 29. The data generation method of claim 28.

30. the first object is an instance of a class; 30. The data generation method of claim 29.

31. the attributes of the first object are specified when the instance is initialized; 31. The data generation method of claim 30.

32. The at least one processor performs the learning process by: Dynamically changing the configuration of the neural network depending on the nature of the data input to the neural network.

32. The data generation method according to any one of claims 22 to 31.

33. The at least one processor performs the learning process by: Dynamically generating a computation graph used for training the neural network; 33. The data generation method according to any one of claims 22 to 32.

34. a processor used for training the neural network and a processor used for generating the data are different processors; 34. The data generation method according to any one of claims 22 to 33.

35. 35. Using the data generated using the data generation method according to any one of claims 1 to 34, a calculation is performed according to the neural network trained using the deep learning framework. Semiconductor integrated circuit.

36. Memory and a control processor; an accelerator; 36. The semiconductor integrated circuit according to claim 35.

37. A program for causing at least one device to execute the data generation method according to any one of claims 1 to 34.

38. Apparatus for carrying out the method of any preceding claim.

39. 35. A system comprising a plurality of devices that perform the data generation method of any one of claims 1 to 34.

40. 35. Data for the semiconductor integrated circuit generated by using the data generation method according to claim 1.

Images