DATA GENERATION METHOD, PROGRAM, DATA GENERATION DEVICE AND SEMICONDUCTOR INTEGRATED CIRCUIT

Abstract

To provide an information processing device that efficiently executes machine learning.SOLUTION: An information processing device according to one embodiment includes: acquisition means that acquires a source code including a code that defines forward processing for each layer configuring a neural network; storage means that stores a correspondence between each forward processing and backward processing corresponding to the forward processing; and execution means that sequentially executes each code included in the source code, and is configured to calculate, at the time of executing each code, on the basis of an input value, an output value of the forward processing defined by each code and also generate, on the basis of the correspondence stored in the storage means, a reference structure for the backward processing in a layer corresponding to the code.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The technology disclosed in this specification relates to machine learning. [Background technology]

[0002] In recent years, machine learning using neural networks has been used in various fields. When performing such machine learning, a developer or the like uses a specific programming language to create source code that defines the network structure of a neural network, and causes a personal computer or the like to execute the source code thus created, thereby enabling the personal computer to perform machine learning (Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yangqing Jia, "Caffe", [online], Berkeley Vision and Learning Center, [Retrieved September 28, 2015], Internet<URL: http: / / caffe.berkeleyvision.org / > Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, there has been a demand for a framework that enables efficient creation of source code that defines the network structure of a neural network. [Means for solving the problem]

[0005] Therefore, various embodiments of the present invention provide an information processing device and an information processing method that efficiently execute machine learning.

[0006] A binary data generation method according to one embodiment is a method for executing, in a semiconductor integrated circuit, an operation corresponding to a neural network trained using a deep learning framework. The method includes generating, based on the neural network, binary data including optimized weights for executing, in the semiconductor integrated circuit, an operation corresponding to the neural network. The optimized weights are information obtained by optimizing the weights of the neural network trained using the deep learning framework after the training. Effect of the Invention

[0007] Various embodiments of the present invention can provide an information processing device and an information processing method that efficiently execute machine learning. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram conceptually showing a conventional technique called "Define-and-Run." [Diagram 2] FIG. 2 is a schematic diagram conceptually showing a method called "Define-by-Run" according to an embodiment of the present invention. [Diagram 3] FIG. 3 is a schematic diagram showing an example of a network configuration of a neural network. [Figure 4] FIG. 4 is a schematic diagram showing another example of the network configuration of a neural network. [Diagram 5] FIG. 5 is a schematic diagram showing yet another example of the network configuration of a neural network. [Figure 6] FIG. 6 is a diagram showing pseudocode for implementing the calculations performed by Linear during forward processing. [Figure 7] FIG. 7 shows pseudocode for implementing the calculations performed by Linear during backward processing. [Figure 8]FIG. 8 is a diagram showing pseudocode for implementing the calculations performed by ReLU during forward processing. [Figure 9] FIG. 9 is a diagram showing pseudocode for implementing the calculations performed during backward processing by ReLU. [Figure 10] FIG. 10 is a diagram showing pseudocode for implementing the calculations performed by Convolution 2D during forward processing. [Figure 11] FIG. 11 is a schematic diagram showing an example of the hardware configuration of a learning device according to an embodiment of the present invention. [Figure 12] FIG. 12 is a block diagram illustrating an example of functions of a learning device according to an embodiment of the present invention. [Figure 13] FIG. 13 is a diagram showing an example of a source code input to a learning device according to an embodiment of the present invention. [Figure 14] FIG. 14 is a schematic diagram conceptually showing the network configuration of a neural network generated by the source code shown in FIG. [Figure 15] FIG. 15 is a diagram showing an example of source code written in Caffe according to the conventional technology. [Figure 16] FIG. 16 is a diagram showing another example of source code input to the learning device according to an embodiment of the present invention. [Figure 17] FIG. 17 is a schematic diagram conceptually showing the network configuration of a neural network generated by the source code shown in FIG. [Figure 18] FIG. 18 is a schematic diagram conceptually showing a network configuration of a neural network generated by source code written in Caffe according to the conventional technology. [Figure 19] FIG. 19 is a diagram showing yet another example of source code input to a learning device according to an embodiment of the present invention. [Figure 20] FIG. 20 is a schematic diagram illustrating step I of the mounting method according to the embodiment of the present invention. [Figure 21]FIG. 21 is a schematic diagram illustrating step II of the mounting method according to the embodiment of the present invention. [Figure 22] FIG. 22 is a schematic diagram for explaining a case where an execution unit based on Python and an execution unit based on a chip communicate with each other. [Figure 23] FIG. 23 is a schematic diagram illustrating step III of the mounting method according to the embodiment of the present invention. [Figure 24] FIG. 24 is a schematic diagram showing a configuration example of a mounting device used in a mounting method (first method) according to one embodiment of the present invention. [Diagram 25] FIG. 25 is a flow diagram illustrating an example of a procedure used in an implementation technique according to an embodiment of the present invention. [Figure 26] FIG. 26 is a schematic diagram showing an operating state of an embedded chip in a mounting method according to an embodiment of the present invention. [Figure 27] FIG. 27 is a schematic diagram showing a configuration example of a mounting device used in a mounting method (second method) according to one embodiment of the present invention. [Figure 28] FIG. 28 is a schematic diagram conceptually showing the functions of a mounting apparatus according to an embodiment of the present invention. [Figure 29] FIG. 29 is a schematic diagram showing a configuration example of a native layer execution unit included in a mounting device according to an embodiment of the present invention. [Diagram 30] FIG. 30 is a diagram showing an example of a structure definition of a multidimensional array module of a mounting device according to an embodiment of the present invention. [Diagram 31] FIG. 31 is a diagram showing mutual compatibility and reference relationships of multidimensional array data in a multidimensional array module of a mounting apparatus according to one embodiment of the present invention. [Diagram 32] FIG. 32 is a diagram for explaining a memory pool module of a mounting device according to an embodiment of the present invention. [Diagram 33] FIG. 33 is a diagram for explaining an example of a structure definition of a memory pool module of a mounting device according to an embodiment of the present invention. [Diagram 34]FIG. 34 is a diagram showing an example of pipelining coding in a mounting device according to an embodiment of the present invention. [Diagram 35] FIG. 35 is a diagram showing the internal state of a virtual machine module in an implementation device according to an embodiment of the present invention. [Diagram 36] FIG. 36 is a diagram showing an example of an execution flow of a virtual machine module in an implementation device according to an embodiment of the present invention. [Figure 37] FIG. 37 is a diagram showing an example of an execution flow of a virtual machine module in an implementation device according to an embodiment of the present invention. [Figure 38] FIG. 38 is a diagram showing an example of address setting in a virtual machine module in an implementation device according to an embodiment of the present invention. [Figure 39] FIG. 39 is a diagram showing a specific example of the overall operation in which the Python layer and the native layer cooperate with each other in an implementation device according to one embodiment of the present invention. [Diagram 40] FIG. 40 is a diagram showing an output of a multiple network configuration in a bytecode generation unit in an implementation device according to an embodiment of the present invention. [Diagram 41] FIG. 41 is a diagram showing an example of code in a bytecode generating unit in an implementation device according to an embodiment of the present invention. [Diagram 42] FIG. 42 is a diagram illustrating an example of the configuration of a native I / F according to an embodiment of the present invention. [Diagram 43] FIG. 43 is a diagram showing an example of a configuration for executing classification and learning by a NN according to an embodiment of the present invention. [Diagram 44] FIG. 44 is a diagram showing an example of a configuration for managing a multidimensional array according to an embodiment of the present invention. [Diagram 45] FIG. 45 is a diagram illustrating an example of the configuration of a data representation conversion unit according to an embodiment of the present invention. [Figure 46] FIG. 46 is a diagram illustrating an example of the configuration of a communication unit according to an embodiment of the present invention. [Figure 47]FIG. 47 is a diagram showing a configuration example of a floating-point and fixed-point execution unit and type conversion unit according to an embodiment of the present invention. [Figure 48] FIG. 48 is a diagram showing an example of a configuration of a memory pool according to an embodiment of the present invention. [Figure 49] FIG. 49 is a diagram showing an example of the configuration of an algorithm execution unit that combines a plurality of NN algorithms according to an embodiment of the present invention. [Figure 50] FIG. 50 is a diagram illustrating a configuration example of a multidimensional array data communication amount reduction unit according to an embodiment of the present invention. [Figure 51] FIG. 51 is a diagram illustrating an example of cooperation with an existing execution unit according to an embodiment of the present invention. [Figure 52] FIG. 52 is a diagram illustrating an example of cooperation with an existing execution unit according to an embodiment of the present invention. [Diagram 53] FIG. 53 is a diagram illustrating an example of the configuration of a bytecode generation unit and a virtual machine according to an embodiment of the present invention. [Figure 54] FIG. 54 is a diagram showing an example of the configuration of a comparison unit according to an embodiment of the present invention. [Figure 55] FIG. 55 is a diagram illustrating an example of the configuration of a function synthesis unit according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Various embodiments of the present invention will now be described with reference to the accompanying drawings, in which the same reference numerals are used to designate the same components in the various drawings. First, in Part 1, we will explain the information processing device according to the embodiment (hereinafter, we will explain it as a learning device, which is an example of an information processing device), and in Part 2, we will explain a method of implementing an algorithm implemented in the information processing device according to the embodiment in an embedded chip (embedded semiconductor integrated circuit).

[0010] Part 1 (Learning device according to embodiment) 1. Background and Overview Machine learning algorithms, including deep learning, can often be formulated as a problem of minimizing the sum of a loss function defined for each model. A loss function is an index that is expressed as the error between the model output and the correct answer for a given learning data sample. Here, the series of processes from inputting data into a model to obtaining the output and comparing it with the correct answer is called a computation graph, and the result is the loss function. The problem of minimizing the loss function can be solved by a common method called gradient method, as long as it is possible to calculate the gradient obtained by differentiating the loss function.

[0011] When trying to implement it as a computer program, one method is to code the loss function and gradients all by yourself, but calculating the gradient of a complex model is generally difficult, and it is often difficult to obtain an explicit formula, so it cannot be written directly as a program. Therefore, the second method is to use a calculation library such as Caffe (http: / / caffe.berkeleyvision.org / ), Torch (http: / / torch.ch / ), or Theano (http: / / deeplearning.net / software / theano / ). The contents disclosed at these URLs are incorporated herein in their entirety by reference.

[0012] In these libraries, you can automatically obtain the gradient function by simply describing the loss function as a combination of the prepared primitives in a dedicated mini-programming language. This is because the gradient of each primitive is defined, so the gradient of the entire combination can be obtained as an automatic differentiation. In other words, even neural networks that can be expressed as large-scale computation graphs, such as those used in deep learning, can be trained by the gradient method using the gradient function if the calculation of the loss function can be explicitly expressed using this mini-programming language.

[0013] Such computational libraries have been based on a computational procedure that the present applicant calls "Define-and-Run." This is an approach in which a computational graph is first defined (Define), the gradient is derived by automatic differentiation, and then learning (Run) is performed using learning data. This approach has the advantage that, when the computational graph does not have complex control syntax (such as if or for) and does not change over time, a series of gradient calculations can be compiled and prepared at high speed as a single unit at the time of Define, and memory management is not required.

[0014] However, in the case of computational graphs with complex control syntax, which have become more common with the development of deep learning research, or models in which computational graphs change dynamically even under meta-conditions that are not dependent on data, there were issues such as low expressiveness of mini-programming languages, difficulty in debugging, and poor memory efficiency due to the inability to dynamically change the structure. Therefore, depending on the complexity of the model and the scale of the data, it was sometimes difficult to implement and execute.

[0015] Therefore, in the embodiment, the applicant proposes a new calculation procedure that the applicant calls "Define-by-Run." Specifically, in the embodiment, instead of having a fixed graph structure in advance as in "Define-and-Run," the applicant adopts an approach in which the graph structure is dynamically extracted and stored in each learning (run), meta-changes are made, and the gradient is recalculated each time.

[0016] This eliminates the need for a mini-programming language that defines the graph in advance, and has the effect of eliminating the costs of design, implementation, and maintenance for developers, and the learning costs and difficulties of debugging for users. In addition, since control syntax can be freely used in common programming languages ​​(C, Java (registered trademark), and Python), neural networks with more complex graph structures can be easily implemented. Furthermore, by enabling meta-change operations based on certain conditions on the graph, memory efficiency is improved and models can be flexibly learned and applied.

[0017] The conceptual difference between the above-mentioned conventional technique called "Define-and-Run" and the technique called "Define-by-Run" according to the embodiment is also clear by comparing FIG. 1 and FIG. 2. FIG. 1 is a schematic diagram conceptually showing the conventional technique called "Define-and-Run", and FIG. 2 is a schematic diagram conceptually showing the technique called "Define-by-Run" according to the embodiment of the present invention. In the Define-and-run configuration shown in FIG. 1, first, a mini-programming language inputs only a model definition and outputs a calculation procedure of a forward (identification) process and a backward (learning) process, which are the substance of a calculation graph (Define step). In the next step, a forward / backward processing system inputs data and updates parameters (weights) according to the calculation procedures of the forward (identification) process and the backward (learning) process (Run step). In contrast, in the define-by-run configuration shown in Figure 2, a general-purpose programming language processing system inputs a model definition, input data, and parameters and executes forward (classification) processing while simultaneously generating a calculation procedure for backward (learning) processing. Here, the model definition is defined in direct compliance with the grammar of the general-purpose programming language, such as function calls, arithmetic operations, loops, and branching. The calculation procedure for backward (learning) processing can also be changed dynamically, independent of the execution of forward (classification) processing. The backward processing system can be called at any time. The backward processing system updates parameters from the input data and the results of forward processing according to the backward calculation procedure.

[0018] 2. Background technology related to neural networks 2-1. Basic processing flow of neural networks The processes performed in a neural network mainly include forward processing, backward processing, and updating of weights. Forward processing refers to the process of processing and propagating information from the input layer to the output layer of a neural network.

[0019] Backward processing refers to the two processes of error backpropagation and weight gradient calculation that are performed from the output layer to the input layer of a neural network. Error backpropagation refers to the process of propagating the error (δ) obtained from the output layer to the input layer. Weight gradient calculation refers to the process of finding the weight gradient (∂W) for a layer that has weights from the error (δ) obtained from the output layer and the output value of the input layer.

[0020] Weight update refers to the process of updating the weights of layers with weights by an algorithm derived from stochastic gradient descent (SGD) using the weight gradient (∂W) obtained by the weight gradient calculation. This weight update is performed once per batch processing unit.

[0021] 2-2. Common computational modules in neural network examples Each layer that constitutes a neural network is realized, for example, by the layer algorithms listed below. -Linear -ReLu -Dropout -Softmax Cross Entropy -Convolution 2D -Pooling (Average Pooling, Max Pooling, etc.)

[0022] Representative examples of weight update algorithms include the following: -momentum-SGD -Adam et al.

[0023] 2-3. Example of neural network configuration (1) Fig. 3 is a schematic diagram showing an example of a network configuration of a neural network. As an example, Fig. 3 shows a neural network in which six intermediate layers (Linear, ReLU, Linear, ReLU, Dropout, and Linear) are arranged between an input layer and an output layer (Softmax). On the page, a right-pointing arrow indicates forward processing, and a left-pointing arrow indicates backward processing. Since the input layer does not have weights to update, the backward processing is performed up to the intermediate layer that has the closest weights to the input layer (in the example shown in Figure 3, the Linear layer located adjacent to the input layer).

[0024] 2-4. Example of neural network configuration (2) Fig. 4 is a schematic diagram showing another example of the network configuration of a neural network. As an example, Fig. 4 illustrates a neural network in which a plurality of (three) intermediate layers (Convolution 2D, ReLU, Convolution 2D, ReLU) are arranged in series between an intermediate layer (Linear) arranged adjacent to an input layer and an output layer (Softmax) and are arranged in parallel. On the page, an upward arrow indicates forward processing, and a downward arrow indicates backward processing.

[0025] 2-5. Example of neural network configuration (3) FIG. 5 is a schematic diagram showing yet another example of the network configuration of a neural network. FIG. 5 illustrates a neural network having a loop (sometimes called a "Recurrent Neural Network") as an example. In the figure, the flow of data in forward processing is indicated by arrows. The intermediate layer (here, Linear) executes a calculation in which the sum of the previous output value of this intermediate layer and the current output value of the input layer is input to this intermediate layer. As a method for realizing backward processing in such a neural network, a method (BPTT) is known in which the network is expanded in the time axis direction in advance and converted into a network without loops.

[0026] 2-6. Layer Algorithm Calculation Contents (Linear) Linear, one of the layer algorithms, performs a calculation to take the weighted average of all nodes in the input layer, repeating the calculation for the number of nodes in the intermediate layer. FIG. 6 is a diagram showing pseudocode for implementing calculations performed by Linear during forward processing, and FIG. 7 is a diagram showing pseudocode for implementing calculations performed by Linear during backward processing.

[0027] 2-7. Layer algorithm calculation content (ReLU) ReLU, one of the layer algorithms, calculates Max(0,val) for each node in the input layer. This algorithm is the most widely used method in recent years for adding nonlinearity (activation function) to neural network calculations. FIG. 8 is a diagram showing pseudocode for implementing calculations performed by ReLU during forward processing, and FIG. 9 is a diagram showing pseudocode for implementing calculations performed by ReLU during backward processing.

[0028] 2-8. Layer algorithm calculation contents (Dropout) Dropout, one of the layer algorithms, randomly selects a certain percentage of nodes and performs a calculation to deactivate the output and backpropagation. This algorithm is not necessary if you are only performing classification (i.e., not performing learning). 2-9. Layer Algorithm Calculation Contents (Softmax Cross Entropy) Softmax Cross Entropy, one of the layer algorithms, corrects the values ​​of the input layer using the following formula.

number

[0029] 2-10. Calculation contents of layer algorithm (Convolution 2D) Convolution 2D, which is one of the layer algorithms, convolves an image with a data structure of Channel*Width*Height. Both the input layer and the output of the layer have a data structure of Channel*Width*Height. This algorithm can also reduce the image size by stride processing. This algorithm also inserts padding into the image of the input layer. In the Channel direction, this algorithm has the same calculation structure as Linear (it repeats the inner product calculation of the input Channel the number of times for the output Channel). FIG. 10 is a diagram showing pseudocode for implementing the calculations performed by Convolution 2D during forward processing. In addition, Convolution 2D performs weight gradient calculation and error backpropagation in the same way as Linear during backward processing. The loop size of each process is the same as that during forward processing.

[0030] 2-11.Layer Algorithm Calculation Contents (Max Pooling) Max Pooling, one of the layer algorithms, shrinks an image vertically and horizontally by taking the maximum value of the image in the input layer. Note that the filter size that takes the maximum value and the stride width of the image shrinking may differ. Also, the number of channels does not change.

[0031] 2-12.Layer Algorithm Calculation Contents (Average Pooling) Max Pooling, one of the layer algorithms, shrinks an image vertically and horizontally by taking the average value of the image in the input layer. Note that the filter size for taking the average value and the stride width for image shrinking may differ. Also, the number of channels does not change.

[0032] 2-13. Weight update algorithm There are various weight update algorithms derived from stochastic gradient descent (SGD), in which the calculations for each weight element are independent. The formula for momentum-SGD mentioned above is as follows:

number

number

[0033] 3. Hardware configuration of the learning device according to the embodiment Next, a hardware configuration of the learning device according to an embodiment of the present invention will be described below. Fig. 11 is a schematic diagram showing an example of the hardware configuration of the learning device according to an embodiment of the present invention.

[0034] 11, the learning device 10 includes a CPU 11, a main memory 12, an input I / F 13, an output I / F 14, a communication I / F 15, an external memory 16, and a user I / F 17, and these components are electrically connected to each other via an internal bus 18. Note that the learning device 10 can also selectively include a GPU (not shown).

[0035] The CPU 11 loads various programs, such as an operating system and a program (a program used to create source code) supporting a programming language (e.g., Python) from the external memory 16 into the main memory 12, and executes instructions included in the loaded programs. The main memory 12 is used to store the programs executed by the CPU 11, and is configured, for example, by a DRAM.

[0036] The input I / F 13 has a function of taking in output data from a measuring device (not shown) and is connected to each component via an internal bus 18. Here, various measurement data output from the measuring device includes information acquired by a sensor or the like, such as temperature, humidity, position information, image data, etc., and may be time-series data such as video data or a temperature data string acquired at a certain interval. The output I / F 14 receives data from each component via the internal bus 18 and outputs it to an output device (not shown) outside the learning device. Here, the data output to the output device may be, for example, control information for driving a motor, or control information for information output devices such as a buzzer, a control switch, an accelerator or brake of an automobile, or a liquid crystal display.

[0037] The communication I / F 15 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these, and is configured to be able to communicate various information with a server device (not shown) via the communication network 20. The external memory 16 is configured, for example, by a magnetic disk drive or a flash memory, and stores various programs such as an operating system and a program that supports a programming language (for example, Python) (a program used to create source code).

[0038] The learning device 10 according to an embodiment having the above configuration can function as a learning device that performs machine learning by having the CPU 11 (and optionally a GPU in addition) execute a predetermined program loaded from the external memory 16 to the main memory 12. For example, the learning device 10 that performs machine learning can be realized as a learning device modeled by a neural network by having the CPU 11 (and optionally a GPU in addition) execute various programs.

[0039] The learning device 10 having the above configuration can be mounted on a corresponding individual (device). The learning device 10 can also be connected to a corresponding measurement device and a corresponding output device. These measurement device and output device may be mounted on the corresponding individual (device), or may be connected as separate devices using communication means.

[0040] In one embodiment, the learning device 10 is any information processing device capable of performing machine learning, including, but not limited to, a personal computer, a tablet, a mobile phone, a smartphone, a mobile information terminal, a touchpad, and an information processing server.

[0041] 4. Functional blocks of the learning device according to the embodiment Next, a brief description will be given of the functions of the learning device 10 having the above configuration. Fig. 12 is a block diagram showing a schematic example of functions of the learning device according to one embodiment of the present invention.

[0042] As described above, the learning device 10 according to the embodiment is based on a method called "Define-by-Run." Specifically, the learning device 10 according to the embodiment is provided with a mechanism for actually executing backward processing and weight update processing by dynamically generating network configuration information required for backward processing and weight update processing at the timing of executing forward processing of a neural network using a general procedural language including branches, loops, and function calls.

[0043] In order to realize such "Define-by-Run", as shown in Fig. 12, the learning device 10 according to one embodiment mainly includes an acquisition unit 110, a storage unit 120, and an execution unit 130. The acquisition unit 110 acquires source code including code defining forward processing of each layer constituting a neural network. Specifically, such source code is created by a developer, a user, or the like using a text editor in a predetermined programming language (e.g., Python, etc.), and the acquisition unit 110 acquires such source code. The acquisition unit 110 can be realized, for example, by the CPU 11, the main memory 12, the external memory 16, the user I / F 17, and the like shown in FIG.

[0044] The storage unit 120 stores a correspondence between each of a plurality of forward processes that can be defined in a source code and a backward process corresponding to the forward process. In the correspondence stored in the storage unit 120, a forward process included in a plurality of forward processes is associated with a corresponding backward process in a one-to-one relationship. That is, in the correspondence stored in the storage unit 120, for example, for a layer (intermediate layer) called Linear, a forward process corresponding to Linear and a backward process corresponding to this forward process are associated. (Such a one-to-one correspondence between a forward process and a backward process is used to execute a process corresponding to the forward process when executing a backward process using a reference structure for backward processing. For example, when a forward process is executed in the order of A→B→C, a backward process is executed in the order of C→B→A, but such a backward process can be realized by implementing both a forward process and a backward process in a pair for each function of A to C.) The storage unit 120 can store various information including the source code acquired by the acquisition unit 110 and various libraries used in the programming language corresponding to the source code. The storage unit 120 can be realized, for example, by cooperation between the CPU 11, the main memory 12, and the external memory 16 shown in FIG.

[0045] The execution unit 130 sequentially executes each code included in the source code acquired by the acquisition unit 110 (stored in the storage unit 120). When the execution unit 130 executes each code, it can calculate an output value of the forward processing defined by the code based on an input value. Furthermore, when the execution unit 130 executes each code, it can generate a reference structure between objects in the layer corresponding to the code. The execution unit 130 can be realized, for example, by cooperation between the CPU 11, the main memory 12, and the external memory 16 shown in FIG.

[0046] Furthermore, in order to realize the above-mentioned "Define-by-Run" technique, the learning device 10 according to one embodiment utilizes three classes, namely, Function, Variable, and Optimizer, by using the above-mentioned acquisition unit 110, storage unit 120, and execution unit 130. Note that the names of these classes are given for convenience and are not limiting. First, the Function class defines a pair of forward and backward operations. This Function class defines the specific layer algorithms exemplified above in "2-6" to "2-12" as subclasses. Next, the Variable class is a class that manages data input and output between Functions. This Variable class has the role of hiding the difference between GPU and CPU, and also has a method (unchain_backward, described later) for aborting backward processing of a network that includes a loop within a finite range. Furthermore, a class called Optimizer is a class that updates the weights.

[0047] 5. Example of operation 1 Next, a specific example of the operation performed by the learning device 10 according to the embodiment having the above configuration will be described. Fig. 13 is a diagram showing an example of source code input to the learning device according to an embodiment of the present invention. Note that the source code illustrated in Fig. 13 has been intentionally simplified for the purpose of explaining the features of the learning device according to this embodiment. Also, the line numbers written on the left side of Fig. 13 are added to explain this specific example, and are not included in the actual source code. In the following, in this embodiment, a case where the source code is written in Python will be described as an example, but the source code may be written in a programming language other than Python. Details of Python are disclosed at https: / / www.python.org / . The contents disclosed at this URL are incorporated herein by reference in their entirety.

[0048] First, a developer or the like creates the source code exemplified in Fig. 13 using a text editor or the like. The source code created in this way is acquired by the acquisition unit 110 (see Fig. 12) of the learning device 10 and stored in the storage unit 120. Next, the execution unit 130 executes each code included in the source code stored in the storage unit 120 line by line. If the source code does not include a control syntax such as an if statement or a for statement as exemplified in Fig. 13, the execution unit 130 executes each line from top to bottom from the first line to the last line. Conversely, if the source code includes a control syntax, the execution unit 130 executes each code in an order according to the control syntax.

[0049] The contents of the source code shown in FIG. 13 will be described. Lines 1 to 3 describe the registration of a Function including parameters using FunctionSet. Specifically, here, Functions including weights (in this example, instances l1, l2, and l3 of the Linear class, which is a Function subclass that defines a layer algorithm that performs an inner product) are registered in an object of a class called FunctionSet. Functions including weights can have their weights updated by the Optimizer. FunctionSet is a mechanism for improving code readability by grouping together Functions updated by the Optimizer.

[0050] Lines 4 and 5 describe the initialization of the Optimizer. Line 4 creates an instance of an Optimizer subclass (a class for updating weights) that implements an algorithm called Adam. The processing of Adam executes the update for each weight element using the formula described in "2-13" above. Line 5 passes a list of Functions containing the weights defined in lines 1 to 3 to the setup method of the Optimizer subclass instance created in line 4. Execution of this setup method initializes the internal state of the Optimizer subclass for updating the weights included in the list of Functions passed to this method.

[0051] Line 6 describes the loading of input data. That is, line 6 illustrates the process of reading input data x and t from a file, etc. In this example, x holds data with a large amount of information, such as an image or sound, and t holds the label ID corresponding to x (data with a small amount of information for answer checking).

[0052] Line 7 describes the input data being held by a Variable object. That is, line 7 creates a Variable class object that holds the input data. The "Define-by-Run" function is realized by the mutual dependency of Variable objects and Function objects, and since arbitrary input data does not have a mechanism for realizing the "Define-by-Run" function, a procedure is required to explicitly hold the input data by an instance of the Variable class.

[0053] Lines 8 to 11 describe the execution of forward processing. Specifically, lines 8 to 11 describe the forward processing by describing it in a general programming language. The "Define-by-Run" function generates a reference structure for backward processing at the same time as executing this definition. By mutually referencing instances of the Function class and instances of the Variable class, it is possible to express the correspondence between any processing and data. This is self-evident because the Variable class represents data and the Function class represents processing. The data structure that expresses the backward calculation procedure shown in Figure 2 is constructed using this reference structure and is defined as the reference structure for backward processing. The reference structure for backward processing grows each time a basic calculation (four arithmetic operations and exponentiation) is performed on a Variable object and a Function is called with a Variable object as an argument or return value. Therefore, even if a forward processing description includes branching, loops, and function calls other than those for Function or Variable, a reference structure for backward processing can be generated. A Function subclass is also associated with each basic calculation on a Variable object.

[0054] Line 12 describes the execution of backward processing. Specifically, line 12 executes backward processing by calling the backward method of the loss variable (instance of the Variable class) obtained as a result of executing the forward processing executed in lines 8 to 11. Backward processing is automatically executed in the reverse order of forward processing by tracing the reference structure for backward processing generated when forward processing was executed.

[0055] Line 13 describes the weight update. Specifically, in line 13, the weight gradient is calculated as a result of executing backward processing in line 12. When the update method of the Optimizer subclass instance is called as in line 13, the weight is updated using this weight gradient. Since the call to the update method for the Optimizer subclass and the call to the backward method of the Variable class are separate functions, it is also possible to execute weight updates after partially executing backward processing. This is effective when you do not want to update the weights of a Function that has already been trained.

[0056] Here, as the contents to be processed during forwarding, attention will be paid in particular to the contents to be processed by the code written on line 8. Line 8 states h1 = F.relu(model.l1(x)).

[0057] When "model.l1(x)" is executed, the following reference structure for backward processing is generated:

number

[0058] After executing "F.relu(", the above reference structure becomes the following reference structure for backward processing.

number

[0059] By the way, we will explain why the Splitter is inserted into the reference structure. If you call “model.l1(x)” again immediately after creating the above reference structure, the following reference structure will be generated.

number

[0060] When propagating the error value from the output layer during backward processing, the splitter instance adds and combines the error values ​​transmitted to x' and x'' respectively, and sets the result as the error of x. By inserting a splitter in this way, it is possible to propagate errors during backward processing from all functions that used x as input during forward processing.

[0061] Next, we will provide additional information on the network configuration of the neural network generated when executing the source code illustrated in Fig. 13. Fig. 14 is a schematic diagram conceptually showing the network configuration of the neural network generated by the source code illustrated in Fig. 13. In Fig. 14, blocks drawn with dotted lines indicate instances of variables, and blocks drawn with solid lines indicate functions.

[0062] First, when line 7 is executed, an instance 30 of variable x and an instance of variable t are generated. For convenience of explanation, only the instance 30 of variable x is shown in Fig. 14, but in reality, an instance of variable t is also generated in the same way. When line 7 is executed, data such as images and sounds is actually held in the instance of variable x.

[0063] Next, when line 8 is executed by the execution unit 130, a neural network is generated in which the instance 30 of the variable x is followed by the function "l1" 31, the function "relu" 32, and the instance 33 of the variable h1. Please note that when line 8 is executed, the execution result of the forward processing described in line 8 is already held by the instance 33 of the variable h1. Also, when line 8 is executed, as described above, a reference structure for the backward processing currently being generated is generated.

[0064] Next, when line 9 is executed by the execution unit 130, a neural network is generated in which the function "l2" 34, the function "relu" 35, and the instance 36 of the variable h2 have been sequentially grown after the instance 33 of the variable h1. Please note that when line 9 is executed, the execution result of the forward processing described in line 9 is already held in the instance 36 of the variable h2. Also, when line 9 is executed, as described above, a reference structure for the backward processing currently being generated is generated.

[0065] Similarly, when line 10 is executed by the execution unit 130, a neural network is generated in which the instance 36 of variable h2 is followed by the function "l3" 37 and the instance 38 of variable y. It should be noted that when line 10 is executed, the execution result of the forward processing described in line 10 is already held in the instance 38 of variable y. Also, when line 10 is executed, as described above, a reference structure for the backward processing currently being generated is generated.

[0066] Finally, when line 11 is executed by the execution unit 130, a neural network is generated in which the instance 38 of the variable y is followed by the instance 40 of the variable loss and the function "Softmax" 39. Please note that when line 11 is executed, the execution result of the forward processing described in line 11 is already held in the new stance 40 of the variable loss. Also, when line 11 is executed, as described above, a reference structure for the currently generated backward processing is generated. When line 11 is executed, the forward processing described in the source code is completed. That is, when line 11 is executed, the difference between the result of the classification performed by the neural network finally obtained and the true classification result given by the variable t is held in the instance 40 of the variable loss. The next step of backward processing is executed with this difference as an input.

[0067] After the forward processing described in lines 8 to 11 is completed, the execution unit 130 then executes line 12, thereby executing the backward processing. Since the reference structure for the generated backward processing has already been generated, the execution unit 130 can calculate the gradient of the weights of each hidden layer (only hidden layers having weights) included in the neural network by executing the backward processing based on this reference structure.

[0068] Next, line 13 is executed by execution unit 130. As a result, the weights of each intermediate layer (only intermediate layers having weights) are updated using the gradient of the weights calculated by executing line 12. That is, learning is executed.

[0069] In this way, in the learning device according to the present embodiment, developers can construct a neural network by describing, line by line, which variable instance is to be given to which function and which variable instance is to hold the execution result obtained by the forward processing. This allows developers to easily describe forward processing intuitively in source code. In addition, developers can describe forward processing in source code (without having to be aware of backward processing) and have the learning device according to the present embodiment execute the source code, thereby allowing the learning device to automatically execute backward processing.

[0070] 6. Comparative Example 1 Next, in order to show the superiority of the learning device according to this embodiment, a case will be described in which a process equivalent to that executed by the source code exemplified in Fig. 13 is written in Caffe according to the conventional technology. Fig. 15 is a diagram showing an example of source code written in Caffe according to the conventional technology.

[0071] As shown in FIG. 15, the definition of a layer (corresponding to Function in this embodiment) is written in a block enclosed in {} written immediately after the term "layer". In the method according to this conventional technology, the dependency between layers must be clearly indicated in the code. For example, the descriptions "top" and "bottom" express the dependency between layers. "Bottom" indicates which layer the input for a layer is obtained from, and "top" indicates which layer the processing result of the layer is output to.

[0072] In the conventional technique, it is necessary to statically define the configuration of the neural network prior to the learning and classification processes performed by the neural network. In other words, it is necessary to define the network configuration of the neural network first, and then execute the learning and classification processes of the neural network. Therefore, it is difficult to dynamically change the network configuration according to the nature of the data. In contrast, in the learning device according to the present embodiment, as described above with reference to FIG. 14, when each code that defines the configuration of the neural network is executed, the forward processing corresponding to that code is executed. That is, the definition of the configuration of the neural network and the execution of the forward processing according to that configuration are executed at the same timing. This makes it easy to dynamically change the configuration of the network according to the nature of the data. For example, a branch may be added to the code in FIG. 13 to switch the layer that executes the forward processing according to the value of the variable t or the data size of the variable x. Also, for example, a variable value may be given as input data instead of the constant "10" on the 9th line of the code in FIG. 19.

[0073] In addition, in the conventional technique, when creating source code, developers, etc., need to write a definition of the network configuration of the neural network so that both forward processing and backward processing can be executed appropriately. In contrast, in the learning device according to the present embodiment, as described above with reference to FIG. 14, developers, etc. do not need to be aware of whether backward processing can be executed appropriately, and after simply writing forward processing (network configuration), the learning device executes the source code, and the learning device automatically executes backward processing. Therefore, developers, etc. can easily and efficiently build a neural network and execute classification and learning.

[0074] Furthermore, in the conventional technique, when creating source code, developers must first define a neural network so that both forward processing and backward processing can be performed appropriately, and then assign data (input data, teacher data, etc.) to the neural network defined in this way. This makes it difficult to intuitively write source code. In contrast, in the learning device according to the present embodiment, a developer writes source code by describing, line by line, which variable instance is to be held in which variable instance when describing forward processing (network configuration) line by line. This allows the developer to write source code intuitively.

[0075] 7. Example 2 Next, another specific example of the operation performed by the learning device 10 according to the embodiment having the above configuration will be described. FIG. 16 is a diagram showing another example of source code input to the learning device according to an embodiment of the present invention. Note that the source code illustrated in FIG. 16 is intentionally simplified for the purpose of explaining the features of the learning device according to this embodiment. Also, the line numbers written on the left side in FIG. 16 are added to explain this specific example and are not included in the actual source code.

[0076] With reference to the source code shown in FIG. 16, it will be described that the learning device according to this embodiment makes it possible to easily construct a neural network using a control syntax (here, a for statement).

[0077] The first to third lines are similar to the first to fifth lines in the source code shown in FIG. 13, and therefore a detailed description thereof will be omitted. The fourth line describes that the processing described in the fifth to tenth lines is to be looped until the value of i becomes 0 to 1000. The fifth and sixth lines are similar to the sixth and seventh lines in the source code shown in FIG. 13, and therefore a detailed description thereof will be omitted. The seventh line describes adding y, which is the processing result of the function l1 and the function relu, to the argument of l1 again. Lines 8 to 10 are similar to lines 11 to 13 in the source code shown in FIG. 13, so a detailed description thereof will be omitted.

[0078] Fig. 17 is a schematic diagram conceptually showing the network configuration of a neural network generated by the source code shown in Fig. 16. In Fig. 17, blocks drawn with dotted lines indicate instances of variables, and blocks drawn with solid lines indicate functions. For convenience of explanation, Fig. 17 shows only the configuration of a neural network generated only when the variable i is 0 to 2.

[0079] As is clear from Figures 16 and 17, in the learning device of this embodiment, even a neural network that repeats the same configuration including variable instances and functions multiple times (here, a configuration in which a function 52 that adds an instance 51 of variable x and an instance 50 of variable y is followed in sequence by a function "l1" 53 and a function "relu" 54, and the output value of function "relu" 54 is held in an instance of variable y) can be easily constructed using a simple control syntax (here, a for statement). In other words, it can be seen that the source code used in the learning device of this embodiment has a high affinity with the control syntax of a programming language.

[0080] 8. Comparative Example 2 Next, in order to show the superiority of the learning device according to this embodiment, a case will be described in which a process equivalent to that executed by the source code exemplified in Fig. 16 is described by using Caffe according to the conventional technology. Fig. 18 is a schematic diagram conceptually showing the network configuration of a neural network generated by source code written by Caffe according to the conventional technology.

[0081] When constructing a neural network similar to those illustrated in Fig. 16 and Fig. 17 using Caffe according to the conventional technology, the developer etc. must first define a basic configuration as shown in Fig. 18, since the configuration of the neural network cannot be defined using a control syntax. Next, the developer etc. must specially describe a process for giving an initial value of an instance 75 of a variable y to a function 72, and for giving an instance 75 of a variable y at the previous time and an instance 71 of a variable x at the current time to the function 72 (the part indicated by the bold arrow in Fig. 18). When constructing a neural network in which the above basic configuration is repeated many times, or when constructing a neural network having a multi-layer structure, the developer etc. must make such a special description each time such a repetition occurs many times, or for each layer in the multi-layer structure.

[0082] In contrast, in the learning device according to this embodiment, the source code can be written easily using the control syntax of a programming language without requiring special writing, as shown in Figures 16 and 17. Therefore, the learning device according to this embodiment can easily and efficiently build even a complex or large-scale neural network.

[0083] 9. Additional Features (1) The learning device according to one embodiment may be capable of performing a function that breaks the reference structure for backward processing. Specifically, when the unchain_backward method of an instance of the Variable class is called, the reference structure for backward processing from the instance toward the input side is severed. For example, assume that the following reference structure for backward processing has been generated by executing forward processing (details of the configuration such as the splitter are omitted).

[0084] A (input layer) ← Convolution2D ← B ← Linear ← C ← Softmax ← D (output layer) Here, A, B, C, and D represent instances of the Variable class, and Convolution2D, Linear, and Softmax represent instances of the Function class.

[0085] In this case, if B. unchain_backward() is called, the reference structure for backward processing is cut off starting from B, and the structure changes as follows: B←Linear←C←Softmax←D(output layer)

[0086] Let us consider a situation where this unchain_backward method is applied to the source code shown in FIG. 16. In this source code, in line 7, y, which is the processing result of the function l1 and the function relu, is added again to the argument of the function l1. In the "Define-by-Run" mechanism, when the description of "x+y" is executed, a copy of y is generated and then the reference structure for the backward processing that has already been generated by the forward processing executed so far is linked. Therefore, in this example, the reference structure for the backward processing continues to grow each time the loop is repeated. The backward processing in line 9 is performed on the reference structure for the grown backward processing. Since the processing in line 9 is included in a loop, the calculation time of the entire loop processing is proportional to the square of the loop size.

[0087] Fig. 19 is a diagram showing yet another example of source code input to a learning device according to an embodiment of the present invention. Note that the line numbers written on the left side of Fig. 19 are added to explain this specific example and are not included in the actual source code.

[0088] By modifying the source code shown in FIG. 16 and periodically calling unchain_backward in line 11 as shown in FIG. 19, the increase in calculation time can be suppressed. Line 9 describes that the processing in lines 10 to 12 is executed every time the loop from line 4 onwards is executed 10 times. Line 11 calls unchain_backward to discard the reference structure for backward processing starting from the loss. This reduces the calculation time for the entire loop.

[0089] By using unchain_backward in this way, even when learning for forward processing that has a loop in the reference structure, excessive growth of the reference structure for backward processing can be suppressed, and the learning process can be performed with a realistic amount of calculations. Furthermore, in another embodiment, it is also possible to use unchain_backward in order to avoid updating the weights for a particular function.

[0090] 10. Additional Features (2) A learning device according to an embodiment can specify a volatile attribute when initializing an instance of a Variable class. When the volatile attribute is valid, a reference structure for backward processing is not generated for forward processing that inputs the Variable.

[0091] When only forward processing is performed using learned weights (i.e., when backward processing does not need to be performed), if the process of generating a reference structure for backward processing is executed during forward processing, waste will occur in both execution speed and memory usage. In such a case, by specifying the volatile attribute when initializing an instance of the Variable class that holds the input data for forward processing, it is possible to stop the generation of the reference structure for backward processing and efficiently execute only forward processing.

[0092] 11. Additional remarks As the most preferred embodiment, an embodiment in which a source code written in Python is input to a learning device has been described, but the technology disclosed in this specification is not limited to the case where a source code written in Python is used. That is, the technology disclosed in this specification is also applicable to the case where a source code written in a programming language equivalent to Python (e.g., R, Julia, Sparkz, MLib, etc.) is used, which can realize at least one of the following: calculating the output value of the forward processing described in each code based on the input value when the learning device executes each code; generating a reference structure for backward processing (and making the backward processing executable based on this reference structure) every time the learning device executes the forward processing described in each code; and defining the configuration of a neural network using a control syntax.

[0093] The techniques disclosed in this specification may be realized by executing source code written in Python or an equivalent programming language, or alternatively, may be realized by executing modules or libraries written in Python or an equivalent programming language.

[0094] In this specification, the names used to identify variables, functions, methods, classes, subclasses, etc. do not limit the technology disclosed in this specification and may be arbitrary.

[0095] The processes and procedures described in this specification can be realized not only by those explicitly described in the embodiments, but also by software, hardware, or a combination of these. Specifically, the processes and procedures described in this specification can be realized by implementing logic corresponding to the processes in a medium such as an integrated circuit, a volatile memory, a non-volatile memory, a magnetic disk, or an optical storage. In addition, the processes and procedures described in this specification can be implemented as computer programs and executed by various computers.

[0096] Although the processes and procedures described herein are described as being performed by a single device, software, component, or module, such processes or procedures may be performed by multiple devices, multiple software, multiple components, and / or multiple modules. Furthermore, although the data, tables, or databases described herein are described as being stored in a single memory, such data, tables, or databases may be stored in multiple memories in a single device or multiple memories distributed across multiple devices. Furthermore, the software and hardware elements described herein may be realized by integrating them into fewer components or breaking them down into more components.

[0097] Part 2 (Methods for implementing algorithms on embedded chips) 1. Background Deep learning is an algorithm that can achieve high performance, but it also requires a large amount of calculations, memory usage, and a large number of learning samples.The spread of GPUs and cloud computing, which provide abundant computing resources at low cost, and web infrastructure, which allows the sharing of learning samples, can be said to have supported the recent rise of deep learning. There are various environments (libraries, frameworks) that support the development of deep learning algorithms. Many development environments have the functionality to improve the learning speed by using GPUs.

[0098] In fields such as fully autonomous driving of automobiles and versatile robot control, advanced information processing capabilities are required to analyze information obtained from various sensors such as cameras and LIDAR (laser ranging) in real time and to control countless motors to solve the problems involved, so there are high hopes for the application of deep learning, which offers performance that sets it apart from conventional technologies. However, these fields rely on embedded environments, which have fewer computing resources than GPUs and the cloud, due to requirements for safety, chip prices, power consumption, etc., and the application of deep learning, which requires high computing resources, has been delayed. The slow application of deep learning to embedded environments is due in part to the fact that the computational resource requirements of these algorithms exceed the performance of realistic and economical embedded environments, as well as the fact that implementations that support deep learning, including software environments, are not yet available. Even in the embedded environment, hardware performance is improving year by year, and deep learning algorithms are also continually being improved to reduce the demands on computing resources, so it is expected that the former factor will gradually be resolved.

[0099] The problem to be solved by the embodiments of the present invention is to develop a framework for designing deep learning algorithms that operate on embedded chips while satisfying product-level requirements, thereby overcoming the remaining barriers to adapting deep learning to embedded environments, mainly in terms of software environments, and accelerating development speed. The learning device according to the embodiment described in Part 1 above is a framework that is GPU-based but provides high productivity in developing deep learning algorithms. It is believed that the optimal means of solving this problem is to expand its functions for embedded environments. Therefore, in the following paragraphs, we will describe the challenges of adapting the learning device according to the embodiment to embedded environments, focusing on the learning device according to the embodiment.

[0100] 2. Issues with the implementation method according to the embodiment The learning device according to the embodiment described in Part 1 above relies on advanced language functions and libraries, so trying to run the algorithms that run on this learning device directly on an embedded semiconductor chip could result in the following disadvantages. First, in terms of security, as libraries and languages ​​get larger, applications become more dependent on implementations that are virtually unknowable, increasing the risk that defects in those implementations will directly result in defects in the chip product. Next, in terms of footprint, the library and language implementation itself puts pressure on the memory resources of the chip product. Furthermore, in terms of overhead, libraries with highly abstracted APIs cannot fully utilize the computing resources of chip products. At least for the large-scale calculations required for neural networks, low-level performance tuning specific to the chip is necessary. For the reasons mentioned above, there is a high possibility that the requirements for product level will not be met if the algorithm that operates in the learning device according to the embodiment is simply operated on an embedded semiconductor chip.

[0101] 3. Concept of the implementation method according to the embodiment In the implementation method according to the embodiment, a new neural network (NN) algorithm designed in a personal computer or the like with abundant computing resources is realized in the shortest possible time in a state where it can operate in any embedded chip (embedded semiconductor integrated circuit) while satisfying product-level requirements. To achieve this, it is desirable for the developer who designs the algorithm and the developer who is deeply aware of the hardware to work as independently as possible. In this embodiment, a technical idea regarding a device (framework) that helps this is proposed.

[0102] 4. Expected development steps for embedded chip development The following three steps are envisioned for development of an embedded chip: Step I: A state in which the code used in the learning device according to the embodiment (for example, the code written in Python) is running on a PC (+GPU) This is the state where algorithms using neural networks with complex configurations can be designed and verified with minimal code writing. This is the concept behind the "Define-by-Run" method mentioned above. Step II: Mixing chip-optimized implementation with Python code This state is one in which the operation and performance of the algorithm designed by the learning device according to the embodiment has been verified on the chip with almost no changes to the Python code. Step III: A state in which the algorithm designed by the learning device according to the embodiment operates only with an implementation optimized for the chip This state is one in which the algorithm operates while satisfying the product-level specification requirements of the chip (i.e., it is capable of highly real-time cooperative operation with other modules and control mechanisms on the chip). In the implementation method of this embodiment, when developing a new algorithm using the learning device of the embodiment, a framework is proposed that enables development to proceed in a short period of time by minimizing the effort of re-modification, re-design, and re-learning between steps I to III above.

[0103] 4-1. About Step I FIG. 20 is a schematic diagram for explaining step I of an implementation method according to an embodiment of the present invention. The configuration shown in FIG. 20 is a configuration that is assumed by the learning device according to the embodiment described in Part 1 above. That is, in this configuration, source code written in Python as one aspect of a programming language uses PyCUDA as one aspect of a library and numpy (BLAS) as one aspect of a library, and these libraries drive a GPU and a general-purpose computer, respectively. Note that "Chainer" shown in FIG. 20 is a name given by the applicant to a framework for writing source code used in the learning device according to the embodiment described in Part 1 above.

[0104] 4-2. About Step II Fig. 21 is a schematic diagram illustrating step II of an implementation method according to an embodiment of the present invention. In the configuration shown in Fig. 21, the front end of Chainer is executed on Python. As shown in Fig. 21, in this embodiment, by providing a native I / F (an interface for calling an implementation equivalent to the main functions of Chainer written in a low-level language such as C), it is possible to execute on a PC and in an optimized state for an embedded chip using the same code.

[0105] Fig. 22 is a schematic diagram explaining the case where an execution unit based on Python and an execution unit based on a chip communicate with each other. As shown in Fig. 22, by providing a communication function in the implementation of the Native I / F, it is possible to remove the dependency on Python from the configuration on the embedded chip (driving the optimized implementation on the embedded chip from Chainer on the PC).

[0106] Native I / F implementation Implement reference code (assuming a low-level language such as C) for Chainer's Function and Optimizer. This reference code will be implemented in a form that does not depend on external libraries such as numpy. Also, implement a memory pool mechanism suitable for dynamic network definition. Also, create a data conversion function with numpy separately from the Function / Optimizer. Furthermore, create floating-point reference code for the above Function / Optimizer. Furthermore, we will create a fixed-point reference code for the above Function / Optimizer. Also, we will create a floating-point / fixed-point data conversion function separately from Function / Optimizer. The reason for creating a fixed-point reference code is that there are quite a few chips that do not have an FPU. Based on the above reference code, we will implement code optimized for various chips.

[0107] 4-3. About Step III Fig. 23 is a schematic diagram for explaining step III of the implementation method according to an embodiment of the present invention. As shown in Fig. 23, a method for outputting a network definition and weights from Chainer as bytecode is added. In addition, a virtual machine is provided that interprets the bytecode and executes neural network processing (forward processing, backward processing, weight update). Chip-optimized implementation of the Native I / F can be reused.

[0108] Configuration 1 (Native IF configuration) FIG. 42 is a diagram illustrating an example of the configuration of a native I / F according to an embodiment of the present invention. A configuration that provides an interface for each NN algorithm that is independent of the type of computer. A processing system that uses an NN algorithm instructs a specific computer to execute the algorithm via this interface. The interface here is a means of defining the input data format, the output data format, and the correspondence between the input data format processing method and the output data format. If the interface is the same, the same output result will be obtained for the same input. For example, there is a function written in C language and its function declaration. There are no particular limitations on the processing system that uses the NN algorithm. For example, existing frameworks for NN design (Chainer, etc.) are available. In addition, processing systems developed in conjunction with the development of algorithms are also available. A computing device, as used herein, refers to a device that performs computations. A computing device is a device that includes a computing core, a memory hierarchy, and the hardware resources required to perform the computations. A general-purpose computer refers to a commonly used computer, such as a computer that easily runs conventional algorithms including the Linux OS and Python. The accelerator here refers to a device that quickly executes specific calculations, including calculations of neural network algorithms. The GPU here is a computer specialized for image processing, but also capable of performing general-purpose calculations. GPUs also include one form of the accelerator mentioned above. Because of the existence of software assets such as CUDA, the ease of implementing NN algorithms is somewhere between that of general-purpose computers and general accelerators.

[0109] Configuration 1-1 (Configuration for performing classification and learning using NN) FIG. 43 is a diagram showing an example of a configuration for executing classification and learning by a NN according to an embodiment of the present invention. The native I / F has at least a forward processing unit. With this configuration, the native I / F can execute the classification process using the NN algorithm. Furthermore, the native I / F has at least a forward processing unit, a backward processing unit, an internal state initialization processing unit for a weight update algorithm, and a weight update processing unit. With this configuration, the native I / F can execute the classification processing and the learning processing by using the NN algorithm. A forward processing unit and a backward processing unit are included for each layer algorithm. An internal state initialization unit for a weight update algorithm and a weight update processing unit are included for each weight update algorithm. Furthermore, the Native I / F has a forward processing call interface and a backward processing call interface for each layer algorithm, and an internal state initialization processing interface for the weight update algorithm and a weight update processing call interface for each weight update algorithm. Furthermore, the implementation that is called through the Native I / F has a Native I / F call management unit. With this configuration, the implementation that is called through the Native I / F can change the implementation that can optimally execute the operation of the Native I / F depending on the difference in the parameters of the Native I / F. Incidentally, if there is no implementation that can execute the operation of the Native I / F, the call management unit of the Native I / F returns an error to the caller. Therefore, the Native I / F The implementation that is called through the I / F can select and execute the implementation that can optimally execute the operation.

[0110] Configuration 1-1-1 (Configuration 1 for performing classification and learning using NN; Configuration for managing multidimensional arrays (multidimensional array management unit)) FIG. 44 is a diagram showing an example of a configuration for managing a multidimensional array according to an embodiment of the present invention. The native I / F further includes a multidimensional array management unit that can perform at least one operation selected from the group including the creation and destruction of multidimensional arrays, acquisition of attributes (number of axes, number of elements per axis), acquisition of aggregation results (sum, average, variance, etc. per axis), and element-by-element arithmetic operations between multidimensional arrays.

[0111] Configuration 1-2 (Configuration for sharing data)

[0112] Configuration 1-2-1 (Configuration 1 for sharing data; data representation conversion section) FIG. 45 is a diagram illustrating an example of the configuration of a data representation conversion unit according to an embodiment of the present invention. Furthermore, the Native I / F has a data representation conversion unit that can convert between a data representation that is dependent on a specific computer (device-dependent data representation) and a data representation that is not dependent on a specific computer (device-independent data representation).

[0113] Configuration 1-2-2 (Configuration 2 for sharing data; + External storage media) Furthermore, the processing system that calls the native I / F has an external storage medium that can store the weight data that has been converted into device-independent data.

[0114] Configuration 1-2-3 (Configuration 3 for sharing data; + Communication unit included) FIG. 46 is a diagram illustrating an example of the configuration of a communication unit according to an embodiment of the present invention. Furthermore, the implementation that is called through the Native I / F has a communication unit. The communication unit can communicate the calling information of the Native I / F to the implementation that is called. Furthermore, when any processing system that uses the NN algorithm attempts to call the Native I / F regardless of whether or not the calling information has been communicated, the implementation that is called through the Native I / F can execute optimal communication processing as necessary. This process makes it possible to hide the physical distance of the computers, the presence or absence of memory sharing, or differences in communication protocols from any processing system that uses the NN algorithm. For example, native I / Fs that are independent of whether or not call information is communicated include an interface for executing a layer algorithm, an interface for executing a weight update algorithm, or an interface for performing data representation conversion.

[0115] Configuration 2 (Native I / F extended version configuration)

[0116] Configuration 2-1 (Configuration 1 of the extended version of the Native I / F; when it has a type conversion unit, and an NN algorithm execution unit for floating point and / or an NN algorithm execution unit for fixed point) FIG. 47 is a diagram showing a configuration example of a floating-point and fixed-point execution unit and type conversion unit according to an embodiment of the present invention. The native I / F has a type conversion unit, and an NN algorithm execution unit for floating point numbers and / or an NN algorithm execution unit for fixed point numbers. For example, there is computer B having only a type conversion unit, computer A having only a floating-point NN algorithm execution unit, or computer C having only a fixed-point NN algorithm execution unit. When such computers A, B, and C are combined with the basic configuration of a Native I / F, the floating-point type data generated by computer A is transferred to computer B. Subsequently, the data transferred from computer A to computer B is converted to fixed-point type data by computer B. Then, the fixed-point type data converted by computer B is transferred to computer C. Then, the fixed-point type data transferred from computer B becomes input data for computer C, and the entire operation of the NN algorithm is executed. These steps can also be executed in the reverse order.

[0117] Configuration 2-2 (Native I / F extended version configuration 2; with memory pool module) FIG. 48 is a diagram showing an example of a configuration of a memory pool according to an embodiment of the present invention. Furthermore, the implementation that is called through the Native I / F has a memory pool module, which can realize dynamic memory management.

[0118] Configuration 2-3 (Native I / F extended version configuration 3: When it has an algorithm execution part that combines multiple NN algorithms) FIG. 49 is a diagram showing an example of the configuration of an algorithm execution unit that combines a plurality of NN algorithms according to an embodiment of the present invention. Furthermore, the native I / F has an algorithm execution unit that combines a plurality of NN algorithms. The algorithm execution unit that combines a plurality of NN algorithms simultaneously executes a plurality of algorithms for frequently occurring combinations of NN algorithms.

[0119] Configuration 2-4 (Native I / F extended version configuration 4; with multi-dimensional array data compression / decompression section) FIG. 50 is a diagram illustrating a configuration example of a multidimensional array data communication amount reduction unit according to an embodiment of the present invention. Furthermore, the implementation called through the Native I / F has a multidimensional array data compression / decompression unit, which is provided in the communication unit.

[0120] Configuration 3 (Native I / F + Chainer execution unit configuration) FIG. 51 is a diagram illustrating an example of cooperation with an existing execution unit according to an embodiment of the present invention.

[0121] Configuration 3-1 (Native I / F + Chainer execution unit configuration 1; with bytecode generator and virtual machine) FIG. 53 is a diagram illustrating an example of the configuration of a bytecode generation unit and a virtual machine according to an embodiment of the present invention. Furthermore, the Chainer execution unit has a bytecode generator, which receives the backward calculation procedure and weights and outputs them as bytecode. For example, the bytecode generator is included in the Python layer of Chainer. The native I / F also includes a virtual machine that interprets the bytecode and executes NN algorithm processing. The NN algorithm processing here refers to any one of forward processing, backward processing, and weight update, or a combination thereof.

[0122] Configuration 3-2 (Native I / F + Chainer execution unit configuration 2; with comparison unit) FIG. 54 is a diagram showing an example of the configuration of a comparison unit according to an embodiment of the present invention. Furthermore, the Chainer execution unit includes a comparison unit that compares input / output results between an existing execution unit and a native layer execution unit that correspond to the same NN algorithm, or compares input / output results between native layer execution units that call native layer execution units of different implementations in the same native I / F.

[0123] Configuration 3-3 (Native I / F + Chainer execution unit configuration 3; with function synthesis unit) FIG. 55 is a diagram illustrating an example of the configuration of a function synthesis unit according to an embodiment of the present invention. Furthermore, the Chainer execution unit has a function synthesis unit, which inputs the backward calculation procedure and replaces a combination of instances of a Function class that can be supported by the "native I / F that executes multiple algorithms simultaneously" with an instance of a Function class that supports the "native I / F that executes multiple algorithms simultaneously." However, if the "native I / F that executes multiple algorithms simultaneously" does not exist in the implementation of the native layer for the computer that executes the backward calculation procedure, the above replacement is not performed. The replacement referred to here can be performed by a partial match search when the backward calculation procedure is regarded as a character string. For example, the function synthesis unit is provided in the Python layer of Chainer.

[0124] Configuration 4 (Configuration of an optimization device that specializes forward processing execution)

[0125] Configuration 4-1 (Configuration 1 of an optimization device specialized for forward processing execution; when having weight optimization processing means) Furthermore, the Chainer execution unit has a weight optimization processing means that executes weight optimization processing suitable for the Function class.

[0126] Configuration 4-2 (Configuration 2 of an optimization device specialized for forward processing execution; with a data memory area reuse means) Furthermore, the Chainer execution unit and the Native I / F have a data memory area reuse unit that reuses the memory area of ​​data input / output between layers. The data memory area reuse unit is provided in the Forward process execution unit or the virtual machine. For example, a flag for identifying that only forward processing is being executed is provided in the arguments of the interface (defined by Native I / F) that executes the forward processing of the virtual machine. The condition for executing this processing is when the volatile attribute is specified for the Variable variable input by the instance of the Chainer's Function class, or when the flag for identifying that only forward processing is being executed is enabled when the forward processing of the virtual machine is executed.

[0127] Effect 1 (Action by NativeIF configuration) This makes it easier to divide the work between developers who design and use NN algorithms and developers who are deeply aware of the computer hardware configuration. For example, developers who design and use algorithms can execute processes on a variety of computers without changing the software that calls the native interface, because the native interface guarantees that the interface for each NN algorithm they want to execute will be uniform. Specifically, it reduces the risk that the software you are developing will become dependent on a specific computer, and as a result, you will be able to select computers based on more fundamental criteria such as the computer's price and its strengths and weaknesses for specific applications. For developers who are deeply conscious of computer hardware configuration, providing an implementation optimized for computers that support the Native I / F will allow a wide range of users of NN algorithms to use the computers they have developed.

[0128] Effect 1-1 (Action by configuration for performing discrimination and learning by NN) Developers who design and use NN algorithms can realize the overall operation of the NN algorithm by calling the interface provided in the NativeI / F using any processing system that uses the NN algorithm. In addition, developers who design and use NN algorithms can realize the overall operation of the NN algorithm by using an implementation that is optimal for the computer they are using, without having to be aware of the specific configuration of the computer.

[0129] Effect 1-1-1 (Action of configuration 1 for performing discrimination and learning by NN; in case of configuration for managing multidimensional arrays (multidimensional array management unit)) Developers who design and use NN algorithms can combine and execute any NN algorithm without going through unnecessary data conversion processes when executing the overall operation of the NN algorithm. At this time, by checking the aggregated results of the contents of the multidimensional array that is the processing result of any NN algorithm, it is possible to check whether the NN algorithm is performing the calculations as intended.

[0130] Effect 1-2 (Action by configuration for sharing data) Effect 1-2-1 (Action by configuration 1 for sharing data; in the case of data representation conversion section) By using device-independent data representation, the data required to realize the overall operation of a NN algorithm can be exchanged between computers with different hardware configurations.

[0131] Effect 1-2-2 It is possible to hide information specific to each computer. (Configuration 2 for sharing data; + External storage media) By converting the weight data into a device-independent data representation and then saving it on an external storage medium, it is possible to execute classification processing on any computer using weights that have been trained using a specific computer.

[0132] Effect 1-2-3 (Action according to configuration 3 for sharing data; + When a communication unit is included) Regardless of the hardware configuration of the computers, the physical distance, or whether or not memory is shared, it is possible to exchange the data necessary to realize the overall operation of the NN algorithm. It is also possible to call an implementation of an NN algorithm implemented in a computer in which the processing system that uses the NN algorithm cannot run from a computer in which the processing system that uses the NN algorithm can run. Therefore, the entire operation of the NN algorithm can be realized using multiple computers connected to a computer network.

[0133] Effect 2 (Effect of the extended version of the Native I / F configuration)

[0134] Effect 2-1 (Action of Configuration 1 of the extended version of the Native I / F; when it has a type conversion unit, and an NN algorithm execution unit for floating point and / or an NN algorithm execution unit for fixed point) In a hardware configuration that combines computers that do not have a floating-point unit (FPU) and those that do, the overall operation of the NN algorithm can be realized using data types appropriate for each computer. The entire operation of the NN algorithm can be realized using floating-point or fixed-point arithmetic. Specifically, computer A transfers floating-point data generated by the floating-point NN algorithm execution unit of computer A to computer B. Next, computer B converts the floating-point data transferred from computer A to fixed-point data using a type conversion unit, and then transfers the fixed-point data to computer C. Computer C transfers the fixed-point data generated by the fixed-point NN algorithm execution unit of computer C to computer B. Next, computer B converts the fixed-point data transferred from computer C to floating-point data using a type conversion unit, and then transfers the floating-point data to computer A.

[0135] Effect 2-2 (Effect of configuration 2 of the extended version of the Native I / F; when the memory pool module is included) When a processing system that relies on a dynamic memory management mechanism calls a native interface, including data generation and destruction, to execute the entire operation of a neural network algorithm, the operation can be realized in a lightweight manner.

[0136] Effect 2-3 (Action of the extended version of Native I / F Configuration 3; when it has an algorithm execution unit that combines multiple NN algorithms) It is possible to avoid unnecessary access to global memory and reduce the overhead of function calls. Therefore, it is possible to execute frequently used combinations of NN algorithms at high speed.

[0137] Effect 2-4 (Effect of Configuration 4 of the Extended Native I / F: When a Multidimensional Array Data Compression and Decompression Unit is Included) When the entire operation of the NN algorithm is executed using multiple computers connected to a computer network, the amount of data communication for the multidimensional array can be reduced. Therefore, the operation speed can be improved.

[0138] Effect 3 (Native I / F + Chainer execution unit action) The overall operation of a NN can be defined and executed by combining algorithms of a NN with support for NativeI / F and algorithms of a NN without support for NativeI / F. As soon as support for Native I / F is available, the entire NN can be operated by replacing it with Native I / F as appropriate. Therefore, there is no need to modify existing software. Even when combining it with a native I / F, you can still enjoy the existing benefits of define-by-run.

[0139] Effect 3-1 (Effect of configuration 1 of Native I / F + Chainer execution unit; when it has a bytecode generator and a virtual machine) The Chainer execution unit has a bytecode generator, and the Native I / F has a virtual machine, which reduces the dependency on advanced libraries and programming languages. Therefore, the entire operation of a NN designed with Chainer can be executed while satisfying product-level requirements even on various computers, including accelerators and other machines with poor execution environments.

[0140] Effect 3-2 (Action due to configuration 2 of Native I / F + Chainer execution unit; with comparison unit) The comparison unit compares the input / output results between an existing execution unit and a native layer execution unit that correspond to the same NN algorithm, and also compares the input / output results between native layer execution units that call native layers with different implementations at the same native I / F. By having such a comparison unit, it is possible to compare the accuracy of the processing result of the NN algorithm execution unit for floating point numbers with the accuracy of the processing result of the NN algorithm execution unit for fixed point numbers. Therefore, it is possible to compare the processing result of the execution unit, which has already been thoroughly tested to be able to correctly calculate the NN algorithm, with the processing result of the newly created native layer. Therefore, it is possible to guarantee that the implementation of the newly created native layer is capable of correctly calculating the NN algorithm.

[0141] Effect 3-3 (Action due to configuration 3 of Native I / F + Chainer execution unit; when function synthesis unit is included) The function synthesis unit inputs the backward calculation procedure and replaces the combination of instances of the Function class that can be handled by the "Native I / F that executes multiple algorithms simultaneously" with an instance of the Function class that has a 1:1 correspondence with the "Native I / F that executes multiple algorithms simultaneously." Note that if the "Native I / F that executes multiple algorithms simultaneously" does not exist, the function synthesis unit does not execute the above replacement. By including this function composition unit in the Python layer configuration of Chainer, the backward calculation procedure is automatically processed regardless of the presence or absence of a "native I / F that executes multiple algorithms simultaneously." If a "native I / F that executes multiple algorithms simultaneously" exists as a result of processing the backward calculation procedure, that native I / F is called and replaced with an instance of the corresponding Function class. This makes it possible to always achieve high-speed overall operation of the NN algorithm. In addition, even for combinations of functions for which there is no "native I / F that can execute multiple algorithms simultaneously," the function composition unit may be beneficial. Specifically, this is the case for the combination of Convolution2D+BatchNormalization when limited to forward processing, or the combination of Linear+BatchNormalization. BatchNormalization is a process that aligns the variance between elements and removes the average for each element of the multidimensional array that is its input, based on long-term statistical information obtained through NN learning. If only forward processing is performed without learning, there is no need to update the variance or average, and it is simply a process that performs a transformation such as y=ax+b for each array element, for example, if a and b are constants. Linear processing is a process that performs matrix multiplication. Convolution2D is a process that calculates a combination of convolution and matrix multiplication. These processes include transformations such as y=ax+b shown above, so by adjusting the weights and biases of Linear and Convolution2D, it is possible to obtain the same results as if the output results of these functions were input to BatchNormalization and processed. By adjusting the weights and biases, the function synthesis unit can convert Convolution2D+BatchNormalization into a single Convolution2D. The same can be done to convert Linear+BatchNormalization into a single Linear.

[0142] Effect 4 (Effect of the configuration of the optimization device specialized for forward processing execution) By reducing the amount of weight information in forward processing or the amount of data memory for input data and executing forward processing, it is possible to reduce memory. Also, by reducing the number of weight elements or executing forward processing without calculating zero weights, it is possible to reduce the amount of calculation.

[0143] Effect 4-1 (Action by the optimization device 1 specialized for forward processing execution; in case of having weight optimization processing means) By having a weight optimization processing means specialized for forward processing execution in the Chainer Function class, it is possible to execute weight optimization processing for any instance of a Function class included in the configuration of a trained network. In this way, by being able to execute weight optimization processing, it is possible to reduce the memory and amount of calculation during forward processing. This allows the entire operation of the NN algorithm to be executed at high speed.

[0144] Effect 4-2 (Action by the optimization device 2 specialized for forward processing execution; in case of having a data memory area reuse means) By giving a flag to execute only forward processing as an argument to the execution unit of the forward processing (Chainer or virtual machine), it is possible to reduce memory during forward processing. This allows the entire operation of the NN algorithm to be executed at high speed.

[0145] 5. Specific steps of the implementation method according to the embodiment The implementation technique according to the embodiment includes a first technique and a second technique. 5-1. First method Fig. 24 is a schematic diagram showing a configuration example of a mounting device used in a mounting method (first method) according to one embodiment of the present invention. As shown in Fig. 24, the mounting device according to one embodiment mainly includes an evaluation board (motherboard) 100 and an embedded chip (embedded semiconductor integrated circuit) 200 detachably mounted on the evaluation board 100.

[0146] The evaluation board 100 mainly includes a CPU 101, a main memory 102, a communication I / F 103, and an external memory 104. These components are electrically connected via an internal bus 109.

[0147] The CPU 101 loads various programs, such as an operating system, from the external memory 103 into the main memory 102, and executes instructions included in the loaded programs. The main memory 102 is used to store the programs executed by the CPU 101, and is configured, for example, by a DRAM.

[0148] The communication I / F 103 is implemented as hardware, firmware, or communication software such as a TCP / IP driver or a PPP driver, or a combination of these, and is configured to be able to communicate with a computer and an input / output device (not shown) operated by a developer, etc., via a communication network (not shown) including Ethernet (registered trademark) or the Internet, etc. The communication I / F 103 is also capable of communicating with a communication I / F 204 (described later) of the embedded chip 200. The external memory 104 is configured, for example, from a flash memory, etc., and stores various programs such as an operating system.

[0149] Next, the embedded chip 200 includes a CPU 201, an accelerator (auxiliary computing device) 202, a main memory 203, a communication I / F 204, and an external memory 205. These components are electrically connected via an internal bus 209. The embedded chip can also selectively include a GPU (not shown).

[0150] The CPU 201 loads source code (e.g., source code written in Python or the like) received from the evaluation board 100 (its communication I / F 103) via the communication I / F 204 into the main memory 203, and executes each code contained in the loaded source code. The accelerator 202 loads source code (e.g., source code written in C language or assembler) received from the evaluation board 100 (communication I / F 103) via the communication I / F 204 into the main memory 203 and executes each code included in the loaded source code. The main memory 203 is used to store the source code executed by the CPU 201 and the accelerator 202, and is configured by, for example, a DRAM.

[0151] The communication I / F 204 transmits and receives various information by communicating with the communication I / F 103 of the evaluation board 100. The external memory 205 is configured by, for example, a flash memory, and stores various data.

[0152] FIG. 25 is a flow diagram illustrating an example of a procedure used in an implementation technique according to an embodiment of the present invention. First, in step 301, a source code written in a first programming language (e.g., Python, etc.) is executed on a personal computer, etc. Based on the execution result, the developer, etc., checks whether the source code runs on the personal computer, etc. Here, the personal computer, etc. refers to a computer having abundant computing resources, and includes, for example, the learning device according to the embodiment described in Part 1 above. The state in which the source code runs on the personal computer, etc. in this step 301 is the same state as step I described in "4-1" above.

[0153] In step 302, the developer causes the CPU 201 of the embedded chip 200 to execute the source code written in Python or the like, which has been confirmed to be operable on a personal computer or the like in step 301, using the evaluation board 100. Based on the execution result, the developer confirms whether the source code is operable by the CPU 201. Note that such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Here, the source code written in Python or the like can be passed to the CPU 201 via the communication I / F 103 of the evaluation board 100 and the communication I / F 204 of the embedded chip 200. If it is found that the source code is not operable by the CPU 201, the developer corrects the source code and repeats step 302. If it is confirmed that the source code is operable by the CPU 201, the developer proceeds to the next step 303.

[0154] In step 303, the developer or the like rewrites (at least a part of) the source code confirmed in step 302 as being operable by the CPU 201 in a second programming language (e.g., C language or assembler) so that it can be executed by the accelerator 202.

[0155] In step 304, the source code rewritten in the C language or the like in step 303 is executed by the accelerator 202 of the embedded chip 200 using the evaluation board 100. Based on the execution result, the developer or the like checks whether the rewritten source code can be operated by the accelerator 202. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Here, the source code written in the C language or the like can be passed to the accelerator 202 via the communication I / F 103 of the evaluation board 100 and the communication I / F 204 of the embedded chip 200. If it is found that the source code cannot be operated by the accelerator 202, the developer or the like corrects the source code and repeats step 304. If it is confirmed that the source code can be operated by the accelerator 202, the developer or the like proceeds to the next step 305.

[0156] In step 305, the evaluation board 100 causes the CPU 201 to compare a result of execution of a first specific code (code to be verified) in a source code written in Python or the like with a result of execution of a second specific code in a source code written in C or the like, the first specific code being rewritten from Python or the like to C or the like (for example, using a module called a unit test executed by the embedded chip 200), and outputs the comparison result. Based on the comparison result, the developer verifies whether the same output is obtained for the same input in both execution results. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Until this verification is completed, the developer repeats the above-mentioned steps 303 to 305. When this verification is completed, the developer proceeds to the next step 306.

[0157] In step 306, the developer or the like tunes the source code written in the C language or the like in step 305 so that the source code runs faster using the accelerator 202.

[0158] In step 307, in the evaluation board 100, the result of the CPU 201 executing the source code written in Python or the like is compared with the result of the accelerator 202 executing the source code written in C or the like tuned in step 306 (for example, by using a module called a unit test executed by the embedded chip 200), and the comparison result is output. Based on the comparison result, the developer verifies whether the same output is obtained for the same input in both execution results. Such an operation can be realized by the CPU 101 of the evaluation board 100 loading and executing a predetermined program stored in the external memory 104. Until this verification is completed, the developer repeats the above-mentioned steps 306 and 307. When this verification is completed, the developer proceeds to the next step 308.

[0159] When step 307 is completed, the embedded chip 200 is in a state where it operates by two source codes, Python or the like and C language or the like. This state will be described with reference to Fig. 26. Fig. 26 is a schematic diagram showing an operating state of the embedded chip in the implementation method according to one embodiment of the present invention.

[0160] As shown in FIG. 26, in the above step 301 (which corresponds to step I), the function caller (i.e., the entity that calls the function) is written in Python or the like, and the callee (i.e., the function to be called) is also written in Python or the like. Next, in the above steps 302 to 307, the calling side of the function is still written in Python or the like, and the called side is in a state where one written in Python or the like and one written in C language or the like are mixed. That is, when step 307 is completed, the embedded chip 200 is in a state where it operates by two source codes, one in Python or the like and the other in C language or the like.

[0161] The ultimate goal of the implementation method of this embodiment is to achieve a state in which both the caller and the called side are written in C language or the like, as shown in the right side of Figure 26, i.e., a state in which the embedded chip 200 operates only by source code written in C language or the like.

[0162] Returning to FIG. 25, in step 308, the developer rewrites all of the source code written in Python that has not yet been rewritten into C language, etc., into C language, etc., so that the embedded chip 200 operates only with source code written in C language, etc. In this step 308, the embedded chip 200 is separated from Python. The source code written in C language, etc., generated in this way, is stored in the external memory 205, etc. of the embedded chip 200. As a result, the embedded chip 200 can read the source code stored in the external memory 205, etc., and execute it in the accelerator 202 to perform machine learning. This state is a target state of the implementation method according to the embodiment, and is a state in which the problems described in "1" and "2" above are solved.

[0163] 5-2. About the second method Fig. 27 is a schematic diagram showing a configuration example of a mounting device used in a mounting method (second method) according to one embodiment of the present invention. The mounting device (Fig. 27) used in the second method is different from the mounting device (Fig. 24) used in the first method in that the embedded chip 200 does not include a CPU. In the second method, the operations performed by the CPU 201 of the embedded chip 200 in the first method are performed by a CPU provided in an external computer (personal computer, etc.) not shown. For example, the computer (personal computer, etc.) referred to here may be the learning device described in Part 1 above (personal computer, etc. exemplified in Fig. 11).

[0164] In the implementation method performed by the second technique, the operations executed by the CPU 201 of the embedded chip 200 in steps 302, 305, and 307 of the implementation method described with reference to Fig. 25 are modified so that they are executed by a CPU provided in the external computer (not shown). To achieve this, the evaluation board 100 shown in Fig. 27 may be communicably connected to the external computer (not shown) via, for example, the communication I / F 103, thereby causing the CPU provided in the computer to execute source code written in Python and receiving the execution results.

[0165] 6. Configuration of mounting equipment Next, a configuration required for the mounting apparatus 100 according to the above embodiment to realize the technique explained in "5" above will be described.

[0166] 6-1. Definitions of terms used to explain the present invention (Difference between classes and modules) A module is a collection of procedures and data defined and implemented to achieve a specific purpose (a concept that is independent of whether it is supported by a specific programming language). Classes are modules defined and implemented with the aid of object-oriented languages ​​such as Python.

[0167] (Python layer and Native layer) The native layer refers to the layer of the native interface and the implementation (software and hardware) called from it. The Python layer refers to the software layer that is expected to be executed on the Python language. Currently, Chainer is written in Python, but it is possible that Chainer will be ported to other programming languages ​​in the future. The functions described here as the Python layer do not necessarily mean that they are specialized for the Python language. The division of roles between the Python layer and the native layer is such that the Python layer is intended for a development environment with a high level of abstraction that is more suitable for algorithm design, while the native layer is intended for a development environment with a low level of abstraction that is more specifically aware of the hardware configuration.

[0168] (Computer and execution unit correspondence) FIG. 52 is a diagram illustrating an example of cooperation with an existing execution unit according to an embodiment of the present invention. The execution part is a method of the Function / Optimizer class that actually calculates the neural network algorithm. The existing execution unit is a general-purpose computer execution unit or a GPU execution unit, or both. The general-purpose computer execution unit calculates the NN algorithm using a general-purpose computer. The GPU execution unit calculates the NN algorithm using a GPU. The native execution unit calculates the NN algorithm using the implementation of the native layer. The native layer is implemented for each type of computer, so it can operate on all types of computers (general-purpose computers, GPUs, accelerators) through the native interface.

[0169] 6-2. Mounting section configuration Fig. 28 is a schematic diagram conceptually showing functions of an implementation device according to an embodiment of the present invention. As shown in Fig. 28, an implementation unit 400 mainly includes a driving unit 401, a Function class / Optimizer class 402, a general-purpose computer execution unit 403, a GPU execution unit 404, a Native layer execution unit 405, a general-purpose computer multidimensional array 406, a GPU multidimensional array 407, a Native multidimensional array 408, and a Variable class 409.

[0170] The driver 401 mainly includes an execution unit that commands the Function class / Optimizer class 402 to execute a certain algorithm (function), and a comparison unit that compares the execution result of the algorithm (function) by the general-purpose computer execution unit 403 (or the execution result by the GPU execution unit 404) with the execution result by the Native layer execution unit 405 using, for example, a module called a unit test, and outputs the comparison result.

[0171] The Function class / Optimizer class 402 causes at least one of a general-purpose computer execution unit 403 , a GPU execution unit 404 , and a Native layer execution unit 405 to execute an algorithm (function) commanded by a driver 401 .

[0172] The general-purpose computer execution unit 403 obtains a multidimensional array corresponding to an algorithm (function) commanded from the Function class / Optimizer class 402 from the general-purpose computer multidimensional array 406, and executes the algorithm (function) using the CPU. The execution result is returned to the drive unit 401 via the Function class / Optimizer class 402.

[0173] The GPU execution unit 404 obtains a multidimensional array corresponding to an algorithm (function) commanded from the Function class / Optimizer class 402 from the GPU multidimensional array 407, and executes the algorithm (function) using the GPU. The execution result is returned to the driver 401 via the Function class / Optimizer class 402.

[0174] The Native layer execution unit 405 obtains a multidimensional array corresponding to an algorithm (function) commanded from the Function class / Optimizer class 402 from the Native multidimensional array 408, and executes the algorithm (function) using the accelerator. The execution result is returned to the driver 401 via the Function class / Optimizer class 402.

[0175] The Variable class 409 holds all multidimensional arrays used by the general-purpose computer multidimensional array 406, the GPU multidimensional array 407, and the native multidimensional array 408, and supplies corresponding multidimensional arrays to the general-purpose computer multidimensional array 406, the GPU multidimensional array 407, and the native multidimensional array 408.

[0176] When the first method described in "5-1" above is adopted as the implementation method, all the components shown in Fig. 28 are arranged in the embedded chip 200 (see Fig. 24). In this case, the general-purpose computer execution unit 403 executes the algorithm (function) using the CPU 201 mounted on the embedded chip 200, the GPU execution unit 404 executes the algorithm (function) using the GPU (not shown) mounted on the embedded chip 200, and the native layer execution unit 405 executes the algorithm (function) mainly using the accelerator 202 mounted on the embedded chip 200.

[0177] On the other hand, when the second method described in "5-2" above is adopted as the implementation method, among the components shown in Fig. 28, the Function class / Optimizer class 402, the general-purpose computer execution unit 403, the GPU execution unit 404, the multidimensional array for general-purpose computer 406, the multidimensional array for GPU 407, and the Variable class 409 are arranged in an external computer (such as a personal computer). In this case, the implementation of the Native layer is still arranged in the embedded chip 200. Furthermore, in this case, the general-purpose computer execution unit 403 executes an algorithm (function) using the CPU of the external computer, and the GPU execution unit 404 executes an algorithm (function) using the GPU of the external computer.

[0178] 6-3. Native layer execution unit configuration Next, a description will be given of the configuration of the above-mentioned native layer execution unit 405. Fig. 29 is a schematic diagram showing an example of the configuration of the native layer execution unit included in the mounting device according to one embodiment of the present invention. As shown in Fig. 29, the Native layer execution unit 405 mainly includes, in the Python layer, a NativeDevice class 501, a NativeArray class 502, a Function class / Optimizer class 503, and a bytecode generation unit 504. Note that the Function class / Optimizer class 503 shown in Fig. 29 is the same component as the Function class / Optimizer class 402 shown in Fig. 28, and the NativeArray class 502 shown in Fig. 29 is the same component as the native multidimensional array 408 shown in Fig. 28. Furthermore, the native layer execution unit 405 mainly includes a device management module 505, a data conversion module 506, a multidimensional array module 507, a function module / optimizer module 508, a virtual machine module 509, and a memory pool module 510 in the native layer.

[0179] The NativeDevice class 502 wraps the device management module in the Native layer in the Python layer and conceals function calls and data input / output to the Native layer. The NativeArray class 502 wraps a multidimensional array in the Native layer in the Python layer. Of the Function class / Optimizer class 503, the Function class wraps the Function module in the Native layer in the Python layer, and the Optimizer class wraps the Optimizer module in the Native layer in the Python layer. Here, the Function class and Optimizer class have been implemented in Chainer and have a function to conceal the difference between execution by a general-purpose computer and a GPU. By extending this function, execution in the Native layer can also be concealed. The bytecode generator generates a bytecode. Details of each of the components illustrated in FIG. 29 will be described later.

[0180] 7. Effects of the Mounting Apparatus According to the Embodiment Deep learning is a developing technology that is the subject of active research and development. It is expected that new layer algorithms with better performance than conventional ones will be invented during the development of embedded chips, creating a demand to incorporate these new algorithms into software or hardware implementations currently under development. In order to realize a neural network configuration including a new layer algorithm that operates in an embedded environment while satisfying product-level specifications, the following development steps must be taken. 1. Implement and verify the algorithm in an environment with abundant computing resources such as GPUs. The algorithm verified in 2.1 is combined with a neural network module that has already been optimized and implemented on an embedded chip, and operation verification is performed. Depending on the results, optimization specialized for the corresponding chip is applied to the algorithm verified in 1. After completing the work in 3.2, the neural network implementation optimized for the chip in question is used alone, and then it is combined with other modules (such as sensors and motor control systems) to verify through various test items whether it meets product-level specifications.

[0181] The implementation device according to the embodiment has a configuration in which, when running a neural network algorithm on Python, it calls, for each layer, an execution unit using the Python language running on a general-purpose computer, an execution unit using a GPU, and an execution unit using an optimized implementation for a specific chip, and is configured to run the entire neural network algorithm using only this optimized implementation for a specific chip via bytecode. Between steps 1 and 2 described in the previous paragraph, the algorithm implementation code created in step 1 can be reused for step 2, and differences in the operation results between steps 1 and 2 can be easily compared. Furthermore, between steps 2 and 3, the optimized implementation created for step 2 can be reused for step 3, and conversely, bug fixes related to the optimized implementation found in step 3 can also be reused for step 2. As a result, it is possible to realize a neural network configuration including a new layer algorithm that operates in an embedded environment while satisfying product-level specifications with minimal development costs.

[0182] Definition of Terms In detailing the embodiments of the present invention, the following terms are defined. The "whole operation" refers to a processing unit that performs a forward process alone or a forward process, a backward process, and a weight update process repeatedly. This whole operation is assumed as an embodiment of learning and classification of a typical neural network.

[0183] 8.Native layer Next, a configuration relating to the native layer of the mounting device according to the embodiment illustrated in FIG. 29 will be described. 8-1.Device Management Module The device management module 505 performs initialization and release processing of devices (software and hardware states that depend on the optimized implementation). Specifically, the processing performed within the device management module 505 differs depending on the type of device, but typical processing includes, for example, allocating and releasing a memory pool, which will be described later. The device does not need to be on the same chip or board as the general-purpose computer that runs Chainer and Python. An optimized implementation that communicates with a device on a different board to initialize and release it is also possible.

[0184] The following is an example of a function definition that initializes or releases a device. (Example 1) Device* chnr_init_device(void) This allows you to initialize the device. (Example 2) void chnr_release_device(Device* device) This allows the device to be released.

[0185] 8-2.Function module The Function module 508 is a group of functions that perform calculations for each layer of the neural network, and defines the following functions for each type of layer. chnr_forward_xxxx(…) -Implemented forward processing (floating point version) chnr_backward_xxxx(…) -Implemented backword processing (floating point version) chnr_forward_xxxx_fx(…) -Implemented forward processing (fixed-point version) chnr_backward_xxxx_fx(…) -Implemented backward processing (fixed-point version) Here, xxxx represents the name given to each layer type. The specific processing contents of each function include those exemplified in "2-6" to "2-12" in Part 1 above.

[0186] 8-3. Multidimensional array (MDArray) module The multidimensional array module 507 manages multidimensional arrays input and output between functions in the native layer. The multidimensional array module 507 can manage any size and number of dimensions. In addition, as described below, the multidimensional array module 507 has a mechanism for mutual conversion with Numpy (a multidimensional array class in the python layer on which Chainer depends) and a multidimensional array library for GPUs. Furthermore, this multidimensional array module 507 can hold not only floating-point type but also fixed-point type. This makes it easy to realize neural network calculations even on hardware that does not have an FPU (floating-point arithmetic unit). This multidimensional array module 507 also has a function for mutual conversion with floating-point multidimensional arrays.

[0187] An example of implementation of the multidimensional array module 507 will be described. FIG. 30 shows an example of a structure definition of a multi-dimensional array module of a mounting apparatus according to an embodiment of the present invention. Next, an example of a function definition is as follows: (Example 1) MDArray chnr_create_md_array(dimensions[], numaxis, type) This allows you to create and initialize multidimensional arrays. (Example 2) void chnr_delete_md_array(MDArray* mdrray) This allows multi-dimensional arrays to be eliminated. (Example 3) void chnr_md_array_add(MDArray* dst, MDArray* a, MDArray* b) This allows addition of elements of multi-dimensional arrays.

[0188] Next, memory management of the multidimensional array in the multidimensional array module 507 will be described. The management (creation and destruction) of the memory area that stores the entities of multidimensional arrays is realized in the native layer. In the case of an embedded environment, it is conceivable that the memory configuration cannot be managed by the memory management mechanism (malloc / free) that is provided as standard by the Linux (registered trademark) OS. Considering the division of roles in the software layers, where algorithm development is carried out in the Python layer and development that is strongly aware of the hardware in the native layer, it is appropriate to implement the management mechanism that is responsible for the characteristics of such hardware environments in the native layer. This memory management mechanism can also be reused when using a virtual machine (when dependency on the Python layer is removed), which will be described later. A class is prepared in the Python layer that wraps the multidimensional array in the native layer, and the timing of memory area creation and release is made to match the instance lifetime of this Python class. This mechanism is necessary to naturally handle multidimensional arrays in Python code. The "Define-by-Run" function also relies on Python's memory management mechanism.

[0189] FIG. 31 shows the mutual compatibility and reference relationships of multidimensional array data.

[0190] 8-4.Memory pool module The memory pool module 510 is a mechanism for reducing the number of calls to a memory management mechanism, which has a high cost (number of processing cycles, etc.), by reusing a memory area that has been secured once. An example of a function definition is as follows: (Example 1) void chnr_momory_pool_init(MemoryPool* momory_pool) This allows the memory pool to be initialized. (Example 2) void chnr_momory_pool_release(MemoryPool* momory_pool) This allows the memory pool to be destroyed. (Example 3) void* chnr_momory_pool_alloc_buffer(MemoryPool* momory_pool, int byte_size, void* old_addr) This makes it possible to secure memory. (Example 4) void chnr_momory_pool_free_buffer(MemoryPool* momory_pool, void*addr) This allows you to free up memory.

[0191] Why is a memory pool module needed in the native layer? (1) Chainer's "Define-by-Run" relies on Python's dynamic memory management mechanism. An example (forward processing of a Linear Function) is shown in Figure 32. In this example, a new instance of Wx (whose contents are a multidimensional array) is created by the statement Wx = x.dot(self.WT) on the third line. Wx is automatically destroyed by Python's memory management mechanism when it is no longer referenced by any variable. The size of the data output by the Function (Wx in the above example) can be changed dynamically depending on the input data size and parameters, and the allocation of the entity (memory area) is also executed within the code flow of forward processing or backward processing. In order to realize "Define-by-Run" (defined at the time of execution of the network configuration), a mechanism is required to allocate memory of the required size at the required timing.

[0192] Why is a memory pool module needed in the native layer? (2) By preparing a class in the Python layer that wraps the multidimensional array in the native layer and making efforts to match the lifetime of the multidimensional array in the native layer with that of the Python layer, it is possible to use the implementation of the native layer while enjoying the flexibility of "Define-by-Run". However, because functions are usually called frequently, calling expensive memory management mechanisms such as malloc and free in the native layer each time can slow down processing speed, which is why it is necessary to provide a function (memory pool) that reuses memory areas that have been allocated once.

[0193] Memory pool implementation example (1) An example of a structure definition is shown in FIG. The processing flow when allocating memory is as follows: 1. Search the buffer_size array for an index where the released flag is 1 and the size from the previous allocation matches the size you want to allocate this time, and if found, set the released flag to 0 and return the buffer_addr value (memory buffer address) for the same index. Here, the released flag is managed by, for example, the sign bit of the buffer_size array element. Searching for array elements using a combination of the size and address from the previous allocation can also reduce address swapping. 2. If a matching index is not found, actually allocate memory (by calling malloc, etc.), add the address and size to the array, and then return the address.

[0194] Memory pool implementation example (2) When freeing memory, the address to be freed is searched for in the buffer_addr array, and if that address is found, the freed flag is set to 1. When releasing a memory pool, memory is released (e.g., by calling the free function) for the elements whose addresses are set in the buffer_addr array.

[0195] Effects of memory pool implementation In most neural networks, the combination of memory sizes is fixed and does not change with each training iteration, so by using the memory pool implementation example above, calls to malloc and other functions can be limited to the first iteration.

[0196] 8-6.Optimizer module The Optimizer module 508 is a group of functions that update weights for each layer of a neural network. The Optimizer module 508 defines the following functions for each weight update algorithm. (Example 1) chnr_op_init_state_xxxx(…) This allows us to implement the internal state initialization process of the weight update algorithm (floating point version). (Example 2) chnr_op_update_one_xxxx(…) This allows us to implement the weight update process (floating point version). (Example 3) chnr_op_init_state_xxxx_fx (...) This makes it possible to implement the internal state initialization process of the weight update algorithm (fixed-point version). (Example 4) chnr_op_update_one_xxxx_fx (…) This allows the weight update process to be implemented (fixed-point version). Here, xxxx represents the name given to each weight update algorithm. It should be noted that the weight update algorithm may include the one described in "2-13" of Part 1 above.

[0197] 8-7. Data Conversion (Converter) Module (1) The data conversion module 506 is a group of functions that convert data formats. An example of a function definition is as follows: (Example 1) chnr_float_to_fixed(MDArray* dst, MDArray* src, int Q) This allows conversion from floating-point to fixed-point. (Example 2) chnr_fixed_to_float(MDArray* dst, MDArray* src) This allows conversion from fixed-point to floating-point types. (Example 3) chnr_host_to_device(MDArray* dst, float* src_data, int src_dimensions[], int src_num_axis, int Q, int async, …) This allows conversion from a device-independent data representation (described below) to a device-dependent data representation (described below). (Example 4) chnr_device_to_host(float* dst_data, int dst_dimensions[], int*dst_num_axis, MDArray* src, int async, …) This allows conversion from a device-dependent data representation to a device-independent data representation.

[0198] Effects of Floating-Point and Fixed-Point Conversion Many embedded semiconductor chips aim to reduce hardware resources (number of transistors and power consumption) relative to the amount of calculation by omitting the FPU (Floating Point Unit) or at least by using a circuit design that does not use the FPU for large-scale parallel calculations. When executing numerical calculation algorithms such as neural networks without using an FPU, a data type called fixed-point type is often used, which expresses numbers including decimal points using integer arithmetic units and shift arithmetic units. The floating-point type is a data type suitable for algorithm design in the sense that it allows real numbers to be handled more intuitively, while the fixed-point type is a data type suitable for effective use of hardware resources. By preparing such conversion functions between data types within the framework for designing and executing neural networks, it is possible to gradually develop neural network algorithms in a unified environment, from the mathematical aspect to the hardware-conscious implementation aspect, while checking the impact of type conversion on a function-by-function basis.

[0199] A device-independent data representation is a data representation that does not have information that is dependent on a specific computer. A typical implementation of this data representation is a multidimensional array in C language format with continuous memory addresses. This type of data representation can be easily handled by using libraries such as numpy in the Python layer, but it is not specific to any library or host language. A device-dependent data representation is a data representation that is suitable for an optimized implementation specialized for a particular computer. By providing functions that convert between these two data representations, it is possible to execute the overall operation by coordinating an optimized implementation that takes hardware into consideration and an implementation that takes algorithms into consideration (for example, easy-to-read code with a structure similar to mathematical formulas written in Python).

[0200] Examples of requirements to consider when converting to device-dependent data representations (1) Memory configuration Should it be in shared memory or in a hardware specific memory area? (2) Memory Alignment Start address, start address for each dimension, padding (3) Byte order Little endian, big endian (4) Data Type Fixed point (Q value) / Floating point, byte width (32bit, 16bit, 8bit, …) (5) Scheduling data I / O in multi-core execution (6) Data communications Communication processing when a device with optimized implementation is on a separate chip or board

[0201] 8-8. Means of communication By applying the following changes to the implementation of the Native layer functions (Native I / F) explained so far, it is possible to execute the entire operation at high speed while communicating with devices on other chips or boards. (1) RPC (remote procedure call) (2) Instruction Queue (3) Reducing the amount of data communication for multidimensional arrays (4) Asynchronous processing of transfer and calculation To explain these change policies, we define the following terms: "Host device": A device that executes the entire operation (in a typical implementation, the device where Chainer code is executed on Python) "Remote device": A device that requires communication processing due to being located on a separate chip or board, etc.

[0202] RPC (remote procedure call) When a function defined in the Native I / F is called, the processing requirements (such as memory allocation and actual calculations) are not directly executed, but rather information representing the processing request (function type and arguments) is generated and sent to the remote device, the remote device executes processing based on the instructions, and then the host device receives the processing results. Command Queue When communicating processing requests via RPC, rather than executing the communication every time a function defined in the Native I / F is called, the information representing the processing request is temporarily stored in a queue (FIFO buffer) to make communication scheduling more efficient. Reducing data traffic for multidimensional arrays Multidimensional arrays have huge data sizes, so reducing the amount of communication is an important issue in improving the overall operation speed. There are two main ways to reduce the amount of communication: (1) Reducing the number of transfers of multidimensional arrays (2) Reducing the amount of data communication for each multidimensional array

[0203] How to reduce the number of transfers of multidimensional arrays The data input and output between layers of intermediate layers (other than the input and output layers of the neural network) and the gradient of the weights only need to exist on the remote device, and communication between devices is not required. The "weights" can also be transferred to the remote device at the initial stage of defining the network structure, and then transferred to the host device when learning is complete. The conversion functions between device-independent data representation and device-dependent data representation described for the data converter module 506 are optimal for managing such transfer timing. Specifically, each function performs the following processing. Data is transferred from a host device to a remote device while converting from a device independent data representation to a device dependent data representation. Data is transferred from a remote device to a host device while converting the data from a device-dependent data representation to a device-independent data representation.

[0204] How to reduce data communication for individual multidimensional arrays A variety of algorithms are known for compressing data. (1) Lossless compression (Huffman coding, run-length compression, etc.) (2) Lossy compression (DCT, scalar quantization, vector quantization, etc.) By specifying the type and parameters of these compression algorithms in the arguments of the function requesting data communication (assuming a conversion function between device-independent data representation and device-dependent data representation), it is possible to reduce the amount of communication using the optimal data compression method based on the characteristics of the data and the accuracy requirements.

[0205] Asynchronous processing of transfers and calculations Many embedded chips have a configuration in which data communication and calculation processing are executed asynchronously by separate hardware. If a function that requests data communication (assuming a conversion function between device-independent data representation and device-dependent data representation) is prepared to execute asynchronously (non-blocking), the entire algorithm can be accelerated by coding that takes this hardware configuration into consideration (a technique generally known as pipelining). The pseudocode for pipelining is shown in Figure 34.

[0206] 8-9. Virtual Machine Module The virtual machine module 509 is a group of functions that interprets bytecode and executes neural network learning / classification processing (forward processing, backward processing, and weight update). The bytecode is assumed to be generated by a bytecode generator in the Python layer, which will be described later, but even if the bytecode is generated by other software, it can be interpreted and executed by the virtual machine module as long as the format is correct. An example of a function definition is as follows: (Example 1) void chnr_init_with_bytecode(VMState* state, char* byte_code) This allows the bytecode to be parsed and the internal state of the virtual machine to be initialized. (Example 2) void chnr_forward(VMState* state) This allows the forward process to be executed. (Example 3) void chnr_backward(VMState* state) This allows the backward processing to be performed. (Example 4) void chnr_update_weight(VMState* state) This allows the weight update process to be performed.

[0207] Bytecode formatting example The following information is stored in binary format: (1) Input / output data information {Number of array dimensions, size, data type (float32, FixedPoint)}*number of variables (2) Weight information {number of array dimensions, size, data type (float32, FixedPoint), realization value}*number of weights (3) Function call information during backward processing {Function type, input / output data index, weight information index, parameters unique to each function type}*number of functions (4) Weight update type and parameters Furthermore, the bytecode may contain an index for the multidimensional array that serves as the input and output of the overall neural network processing. By storing this index in the bytecode, the user code that uses the virtual machine can properly link the multidimensional array that serves as the input to the overall neural network processing and the multidimensional array that serves as the output in terms of function calls. For example, this linking can be performed using the following flow: (Step 1) The user code obtains an input multidimensional array for the entire process by calling a function provided in the virtual machine configuration. (Step 2) The user code copies the input data into the multidimensional array obtained in step 1. (Step 3) User code calls functions that perform the overall operations provided by the virtual machine configuration. (Step 4) The user code obtains the output multidimensional array of the overall processing by calling a function prepared in the virtual machine configuration (this multidimensional array stores the processing results of the overall operation executed in step 3 above. The functions in steps 3 and 4 above do not necessarily need to be separate and may be a single function). (Step 5) The user code obtains the contents of the output data from the multidimensional array obtained in step 4.

[0208] Example of implementation of process flow for initializing the internal state of a virtual machine (1) Interpret the "input / output data information" in the bytecode and generate a list of multidimensional arrays to be input and output by the Function. (2) Interpret the “weight information” in the bytecode to generate a list of weights and weight gradients (both multidimensional arrays). (3) Interpret the "Function call information for backward execution" in the bytecode and generate a list of structures (FunctionState) containing the following information for forward and backward execution, respectively (ID of the function executing the forward / backward processing, addresses of input / output data, addresses of weights, addresses of weight gradients, and parameters specific to each type of Function). (4) Interpret the “weight update type and parameters” in the bytecode and initialize a multidimensional array of the internal state of the weight update algorithm and a structure (OptimizerState) that contains the following information (address of the function that performs the weight update, address of the weight, address of the weight gradient, internal state of the weight update algorithm, and parameters specific to each weight update type).

[0209] The internal state of the virtual machine is shown in Figure 35.

[0210] Execution Flow Example (1) of Virtual Machine Module (Forward Processing and Backward Processing) The virtual machine module executes processing such as the pseudo code shown in FIG.

[0211] Virtual machine execution flow example (2) (Optimizer) The virtual machine module executes a process such as the pseudo code shown in FIG.

[0212] Configuration 2 of an optimization device specialized for forward processing execution; Case where a data memory area reuse means is included (1) When performing the overall operation, if you only want to perform the classification process without learning (weight update), you only need to perform the forward process. In this case, the following data is not required. (1) Data input / output between layers that is not accessed by the currently executing function (2) Gradient of weights (3) The internal state of the weight update algorithm When initializing the internal state of the virtual machine, the weight gradient and the internal state of the weight update algorithm do not need to be allocated. The amount of memory allocated for data input and output between layers can also be reduced by, for example, following the procedure described in the next paragraph.

[0213] Configuration 2 of an optimization device specialized for forward processing execution; Case with data memory area reuse means (2) Example of procedure for initializing the internal state of the virtual machine module (1) Calculate the sum of the data size (memory size) input and output for each function and select the one with the largest size. (2) When initializing the structure that handles multidimensional arrays (MDArray), set the address so that the memory area allocated in 1 can be reused. By setting addresses for each layer so that the left and right ends of the memory area are alternately used as input and output, copying of array data is prevented. If a function that performs input and output with a loop is included, output data carried over to the next iteration is excluded from the reuse shown in this procedure and memory area is allocated separately.

[0214] Configuration 2 of an optimization device specialized for forward processing execution; Case with data memory area reuse means (3) An example of address setting for data input / output by a function is shown in FIG.

[0215] A note about bytecode formatting In the explanation so far, we have shown an example of simply implementing the information stored in the "Function call information during backward processing" in ascending or descending order. However, by storing multiple execution orders in the bytecode, or by storing repetition and branching instructions in the bytecode, it is possible to execute more advanced processing, such as dynamically changing the configuration of the neural network according to the nature of the input data when the virtual machine is running. The memory management mechanism described above for the memory pool module can be used to realize such a dynamic mechanism.

[0216] Linking data input / output from the virtual machine to external code The "virtual machine internal state initialization process" creates a list of data to be input and output between Functions, and the simplest way is for external code that calls virtual machine functions to directly access the elements of this list. If the variable name of the Python Variable instance is stored in the "input / output data information" when the bytecode is generated, this name can also be used to link input and output.

[0217] 9.Python layer Next, a configuration relating to the Python layer of the mounting device according to the embodiment illustrated in FIG. 29 will be described. 9-1.NativeArray class The NativeArray class 502 is a class that wraps a multidimensional array in the Native layer in the Python layer. The NativeArray class 502 is generated as an instance that has a 1:1 correspondence with the multidimensional array in the Native layer. The NativeArray class 502 also has a lifetime management function using reference counting as a basic function of a Python object. Furthermore, the NativeArray class 502 has a function to request the release of a multidimensional array in the Native layer when its lifetime ends. Also, the NativeArray class 502 has a copy of type information of the multidimensional array in the Native layer, and has a function of transmitting the copy to other objects in the Python layer. Furthermore, the NativeArray class 502 has functions such as data copying and addition for each array element, and has a function of requesting the execution of these functions to the Native layer. In addition, the NativeArray class 502 has the ability to operate in a compatible manner with other Python-layer multidimensional array libraries on which Chainer depends, such as Numpy and GPUArray.

[0218] 9-2.NativeDevice class The NativeDevice class 501 is a class that abstracts the optimized implementation and reference implementation of the Native layer. The NativeDevice class 501 has a function of requesting the following processes to the Native layer in response to requests from other objects in the Python layer. (1) Initializing and releasing the device (2) Creating and copying multidimensional arrays (creating a NativeArray instance that wraps this in a Python layer) (3) Conversion between device-independent and device-dependent data representations (conversion between floating point and fixed point can also be specified) (4) Executing Function and Optimizer processing (calling individual functions in the native layer)

[0219] 9-3.Function class The Function class 503 is a class that defines a pair of forward processing and backward processing. The Function class 503 is a class that exists in Chainer, but adds a function to request forward processing and backward processing to the native layer. Method implementation example is as follows. (Example 1) forward_native(...) This allows forwarding processing to be requested to the native layer. (Example 2) backward_native(...) This allows backward processing to be requested to the native layer.

[0220] Assumed processing flow when calling forward_native or backward_native (1) Calculate the output data size from the input data size and the parameters at the time of initializing the Function instance. (2) The output data size calculated in (1), the input data instance (NativeArray), and the Function classification (Linear, ReLU, …) are passed to the NativeDevice instance to request a function call in the Native layer. (3) In response to this call, the NativeDevice instance performs the following operations: (A) Ask the native layer to generate a multidimensional array of output data. If the function overwrites the input data, this process is not performed. (B) Determine the native layer function to be called based on the input data type (floating point, fixed point) and the Function category, and call it (the native layer function writes the processing results to the multidimensional array allocated in A). (C) Create a NativeArray instance that wraps the multidimensional array allocated in (A) above. (4) The NativeArray instance created in (C) above is returned as the return value of the Function.

[0221] 9-4.Optimizer class The Optimizer class 503 is a class that updates weights. The Optimizer class 503 is a class that exists in the Chainer, but adds a function of requesting state initialization and weight update processing to the native layer. Method implementation example is as follows. (Example 1) init_state_native(...) This makes it possible to request the native layer to perform internal state initialization processing for the weight update algorithm. (Example 2) update_one_native(...) This allows the weight update process to be requested from the native layer. The processing flow when these methods are called is the same as that explained in the "Function Class" above.

[0222] 9-5. Example of overall operation linked with native layer A specific example is shown in FIG.

[0223] 9-6.Bytecode generation (output) part The bytecode generator 504 is a mechanism that converts the network configuration of the neural network defined in "Define-by-Run" into bytecode (an interpretable and executable data format) and outputs it. The bytecode format can be, for example, the one mentioned above for the "Virtual Machine Module". However, in addition to this format, output in the following formats can also be considered. (1) Caffe's neural network definition format It can be run on Caffe (Caffe is one of the leading frameworks for designing and executing neural networks). (2) Programming languages ​​such as C and Java (registered trademark) It is possible to generate software that performs the overall operation. (3) Hardware description languages ​​such as HDL and Verilog It is possible to synthesize hardware that performs the entire operation.

[0224] An example of a function definition in the bytecode generator is as follows: Function name: write_network_difinition(output_node, path, format) Function specification: The network configuration connected from the output_node to the input side is output to the file specified by path in the format specified by format. The output_node can be specified as a list (multiple nodes can be the starting point).

[0225] Example of a procedure to output bytecode from a reference structure for backward processing As explained in Part 1 above, Chainer has the ability to generate a reference structure for backward processing according to a description of the natural forward processing calculation. Since forward processing can be calculated by tracing the reference structure for backward processing in reverse order, if bytecode is generated from this reference structure, both forward and backward processing can be executed. This procedure can be broadly divided into the following steps: (1) Generate element information for creating bytecode Input / Output Data Information Generation Weight information generation Generates function call information during backward processing (2) Converting element information into bytecode

[0226] Element information generation procedure for creating bytecode -The "reference structure for backward processing" is traced starting from the output_node passed to write_network_definition and the following processing is performed. (1) If the current node is a Variable, the information about the multidimensional array (size, number of dimensions, floating point / fixed point (Q value)) is added to the "Input / Output Data Information" list. (2) If the current node is a Function, perform the following processing. (i) Information about the multidimensional weight array (size / number of dimensions, floating point / fixed point (Q value), weight realization value) is added to the “weight information” list without allowing duplicates (as multiple Function instances may share the same weight). (ii) Add the function type, the index of the input / output data, the index of the weights, and parameters specific to each function type to the list of “Function call information during backward motion.” If multiple starting nodes are passed to output_node, the procedure in the next paragraph will be used to avoid duplicate registration of the same node.

[0227] Steps for creating element information when multiple starting nodes are passed (1) Create an empty list of "Function call information during backward processing." (2) Perform the following steps for each starting node in the output_node. (A) Create a list of "function call information during backward processing" specific to the origin node. (B) The registration procedure described in the previous paragraph is performed on the list created in (A) above. At this time, the registration procedure is not performed on nodes that are already registered in the list created in (1) above to avoid duplicate registrations. (C) Concatenate the list created in (A) above to the front of the list created in (1) above.

[0228] Python Layer: Bytecode Generator (6) Converting element information into bytecode Convert the following information created in the procedure "Generating element information for creating bytecode" into the format specified by the format argument of write_network_difinition. (1) Input / output data information generation (2) Weight information generation (3) Function call information generation during backward processing As the format, the example given above for the "bytecode generator" can be used.

[0229] Multiple network configuration output The write_network_definition function described above about the "bytecode generator" is designed to write the network configuration directly to the file passed in the argument path, but you can also pass an object to this path argument to write the configurations of multiple networks to bytecode. Here, the network configuration refers to the components (1), (2), and (3) explained in "Python Layer: Bytecode Output Unit (6) Conversion of Element Information to Bytecode". This "object for writing multiple network configurations to bytecode" shares the same "(2) weight information" between "multiple network configurations" and reduces the amount of weight information written to the bytecode. "(1) Input / output data information generation" and "(3) Function call information generation during backward processing" are generated independently using the steps described above, even if there is partial overlap in the information. Figure 40 shows an example of code when using this object. In this code example, on line 6, the reference structure for backward processing is traced from nodeA and the network configuration is output, and on line 7, the reference structure for backward processing is traced from nodeB and the network configuration is output. On line 8, these two network configurations are output to a single file (. / bytecode.bin).

[0230] How to specify different function invocation orders for forward and backward processing on the same network As explained in Part 1 above, Chainer has a function (unchain_backward method) that terminates the backward reference structure that starts from a specific Variable instance and goes to the input layer. By combining this unchain_backward method with the "output of multiple network configurations" explained in the previous paragraph, it is also possible to specify different function invocation orders for forward and backward processing calculations in the same network. In the code example shown in Fig. 41, calling #1 outputs a network definition that executes all processes from A to D, whereas calling #2 outputs a network definition that executes only processes from B to D. When the virtual machine executes the bytecode, it is possible to distinguish between them, for example, by performing forward processing on the network configuration output by #1 and backward processing on that of #2.

[0231] 10. Configuration common to the native and Python layers 10-1. Algorithm execution unit that combines multiple NN algorithms In general neural network configurations, there are frequent combinations of functions. (Example 1) Linear → ReLU, Linear → ReLU → Linear → ReLU (Example 2) Convolution2D→ReLU, Convolution2D→ReLU→Convolution2D→ReLU By defining a combination of such frequently occurring functions as a single function and implementing it specifically for that calculation in both the Python layer and the native layer, we can enjoy the following benefits in terms of both algorithm design and hardware execution efficiency. (1) The overhead required to call a function (function calls and communication) can be reduced. (2) High execution efficiency can be achieved by implementing functions that take into account data dependencies and parallelism across multiple functions (e.g., by making effective use of cache memory and reducing the amount of data that needs to be directly accessed in main memory). (3) By using more abstract functions when designing algorithms, it becomes easier to understand and define complex network configurations.

[0232] In general, while the speed of the arithmetic core of modern computers has improved significantly, the memory access performance has not improved that much. Therefore, when looking at the performance of the entire computer, there is a problem that sufficient calculation performance cannot be achieved because it is limited by memory access. To solve this problem, a mechanism is used to fill the speed gap between the two by placing small-capacity but particularly fast memories called cache memories and register files physically close to the arithmetic cores and performing many calculations on the cache memories.

[0233] By the way, the following combination of functions is frequently seen in the configuration of neural networks. ·Convolution2D→ReLU ·Convolution2D→ReLU→Convolution2D→ReLU Convolution2D requires a large amount of calculations relative to the size of the input and output data, so there is a high chance that the cache memory and other mechanisms can be effectively utilized to demonstrate the performance of the calculation core. In contrast, ReLU requires a small amount of calculations relative to the size of the input and output data, so this opportunity is small. When Convolution2D and ReLU are executed as separate functions, all the data resulting from Convolution2D must be written to main memory, and then the contents must be transferred to the peripheral computing core again to perform the ReLU calculation. This is because it is not clear whether ReLU will use the results of Convolution2D immediately after the processing is completed.

[0234] Therefore, if Convolution2D and ReLU are executed as a single function, the results of Convolution2D processing can be used directly as input for ReLU processing in cache memory or register file before being written out to main memory, thereby reducing the frequency of data transfer to main memory and increasing the chances of executing processing more efficiently (faster). If more functions can be executed as a single function, such as Convolution2D→ReLU→Convolution2D→ReLU, there will be more opportunities to improve processing efficiency. This is because the amount of access to main memory can be reduced more aggressively by taking into account the cache memory size and data dependencies within the combination of functions.

[0235] 10-2. Configuration 1 of an optimization device specialized for forward processing execution: In case of having weight optimization processing means Some neural network layer algorithms can reduce the amount of calculations and memory usage by optimizing the weights specifically for cases where only forward processing is performed without backward processing. The reasons why such optimization is possible are as follows. To train a neural network using stochastic gradient descent, the weight vector needs to have high numerical precision and a high degree of freedom in the range of values. This is because small value updates must be accumulated during training, and the range of values ​​in which the weight vector will change cannot be fully predicted in advance. In contrast, if only forward processing is performed, such precision and freedom are not necessary, and memory and calculation volume can be reduced by reducing the amount of weight information before performing forward processing. The amount of calculation can be reduced by reducing the number of weight elements and not calculating weights that are 0. For example, in linear (inner product between layers) processing, a technique is known that compresses the weight data size and reduces the calculation volume by singular value decomposing the weight information (matrix of the number of input nodes * the number of output nodes) and deleting elements that make the diagonal components smaller. (J. Xue, J. Li, and Y. Gong. Restructuring of deep neural network acoustic models with singular value decomposition. In Interspeech, 2013) By adding a method to the Chainer Function class that performs such forward-specific weight optimization processing, it is possible to reduce the computational resources required when only forward processing is performed using weights that have already been trained. Like the existing forward and backward methods in the Function class, this method has the ability to hide differences in hardware implementation (general-purpose computers, GPUs, native) depending on the type of multidimensional array of weights held by the Function (calling different reinitialization implementations).

[0236] Software and hardware configuration supplement Up to now, the details of the embodiment have been described by exemplifying that specific functions are subordinated to functions and classes in the Python layer and the Native layer. However, the division of roles of these software layers, classes, and functions is merely an example for specifically explaining the configuration of functions according to the embodiment of the present invention, and as shown in the following example, it is possible that each function according to the embodiment of the present invention is implemented in a class, layer, or hardware different from that described above. (1) The processing contents described in "Configuration 2 of an optimization device specialized for forward processing execution; when a data memory area reuse means is included" can be executed in advance by a bytecode generator rather than a virtual machine module. (2) The function described in “Function combining multiple functions” can be implemented using specialized hardware (FPGA, ASIC) rather than through software-level optimization. Therefore, the configuration according to the embodiment of the present invention does not directly depend on the functions or classes of the Python layer or the Native layer, or on implementation based on software. [Explanation of symbols]

[0237] 10 Learning Device 100 Evaluation Boards 110 Acquisition Department 120 Storage section 130 Executive Department 200 Embedded chips (embedded semiconductor integrated circuits) 401 Drive unit 402 Function class / Optimizer class 405 Native layer execution unit Multidimensional array for 408 Native 409 Variable Class 504 Bytecode Generator 505 Device Management Module 506 Data Conversion Module 507 Multidimensional Array Module 508 Function module / Optimizer module 509 Virtual Machine Module 510 Memory Pool Module

Claims

1. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the at least one processor obtaining the optimized weights by reducing a number of elements of the weights of the neural network; The at least one processor reduces the number of elements based on the magnitude of a diagonal element when performing singular value decomposition using weights of the neural network; the data is in a data format that can be interpreted and executed in the semiconductor integrated circuit; Data generation method.

2. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, The data includes weights of other neural networks and is in a data format that can be interpreted and executed by the semiconductor integrated circuit. Data generation method.

3. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the optimizing includes optimizing for operations executed on the semiconductor integrated circuit; the optimized weights are weights optimized for a forward process executed in the semiconductor integrated circuit, the data is in a data format that can be interpreted and executed in the semiconductor integrated circuit; Data generation method.

4. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the data includes information regarding a call of a function and is in a data format that can be interpreted and executed by the semiconductor integrated circuit; The information regarding the invocation of the function includes information regarding the invocation of a third function that integrates the functions of the first function and the second function used in the deep learning framework, Data generation method.

5. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the data includes information regarding a call of a function and is in a data format that can be interpreted and executed by the semiconductor integrated circuit; The at least one processor obtains information about the invocation of the function based on a reference structure generated by training the neural network using the deep learning framework. Data generation method.

6. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, The data includes an index of a multidimensional array that is an input / output of the neural network, and is in a data format that can be interpreted and executed in the semiconductor integrated circuit. Data generation method.

7. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, The network configuration of the neural network is defined using a Define-by-Run function in the deep learning framework, the data is in a data format that can be interpreted and executed in the semiconductor integrated circuit; Data generation method.

8. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the data includes information on network configurations of a plurality of neural networks including the neural network, and is in a data format that can be interpreted and executed by the semiconductor integrated circuit; Data generation method.

9. A data generation method for executing, in a semiconductor integrated circuit, a calculation according to a neural network trained using a deep learning framework, comprising: At least one processor generates data including optimized weights based on the neural network for executing a calculation according to the neural network in the semiconductor integrated circuit; The optimized weights are information obtained by optimizing weights of the neural network trained using the deep learning framework after the training, the data does not include information used only for executing backward processing, and is in a data format that can be interpreted and executed by the semiconductor integrated circuit; Data generation method.

10. The method of claim 1, wherein the at least one processor obtains the optimized weights by compressing a data size of the weights of the neural network. The data generating method according to any one of claims 1 to 9.

11. The method of claim 10, wherein the at least one processor obtains the optimized weights by reducing a number of elements of the weights of the neural network. A data generating method according to any one of claims 2 to 9 or claim 10 without relying on claim 1.

12. The method according to claim 11, wherein the at least one processor reduces the number of elements based on the magnitude of diagonal elements when singular value decomposition is performed using weights of the neural network. The data generating method according to claim 11.

13. The data includes weights of other neural networks. A data generating method according to claim 1 or any one of claims 3 to 12 not relying on claim 2.

14. The method of claim 13, wherein the at least one processor obtains the optimized weights by sharing weights between the neural network and the other neural network. The data generating method according to claim 2 or 13.

15. The optimized weights include weights expressed in fixed point. The data generating method according to any one of claims 1 to 14.

16. The optimization includes optimizing for operations executed on the semiconductor integrated circuit. A data generating method according to claim 1, claim 2, or any one of claims 4 to 15 not relying on claim 3.

17. the optimized weights are weights optimized for forward processing executed in the semiconductor integrated circuit; 17. The data generating method according to claim 16.

18. the data includes information regarding invocation of a function; Any one of claims 1 to 3, or A data generating method according to any one of claims 6 to 17, which does not cite claim 4 or claim 5.

19. The information regarding the invocation of the function includes information regarding the invocation of a third function that integrates the functions of the first function and the second function used in the deep learning framework, 20. The data generating method according to claim 18.

20. the first function and the second function are functions related to a layer algorithm; 20. The data generating method according to claim 4 or 19.

21. The first function and the second function are either a matrix multiplication function, a convolution function, or an activation function.

21. The data generating method according to claim 4, claim 19 or claim 20.

22. The method according to claim 21, wherein the at least one processor obtains information about the invocation of the function based on a reference structure generated by training the neural network using the deep learning framework.

22. The data generating method according to claim 4 or any one of claims 18 to 21.

23. The reference structure is a data structure including information on an execution order in a forward process or a backward process of the neural network.

23. The data generating method according to claim 5 or claim 22.

24. The reference structure is a data structure generated using a Define-by-Run function in the deep learning framework.

24. A data generating method according to claim 5, claim 22 or claim 23.

25. The data includes indexes of multidimensional arrays that are input and output to the neural network. A data generating method according to any one of claims 1 to 5 or any one of claims 7 to 23 not relying on claim 6.

26. The network configuration of the neural network is defined using a Define-by-Run function in the deep learning framework. A data generating method according to any one of claims 1 to 6 or any one of claims 8 to 24 not relying on claim 7.

27. The neural network is trained using a programming language at a higher level than C.

27. A data generating method according to any one of claims 1 to 26.

28. The programming language is the Python language.

28. The data generating method according to claim 27.

29. 29. The data generating method according to claim 1, wherein the data includes information regarding a network configuration of the neural network.

30. The data includes information regarding network configurations of a plurality of neural networks including the neural network. A data generating method according to any one of claims 1 to 7 or any one of claims 9 to 28 not relying on claim 8.

31. The data does not include information used solely for performing backward processing. A data generating method according to any one of claims 1 to 8 or any one of claims 10 to 30 not relying on claim 9.

32. The information used only for performing the backward process includes at least the gradient of the weights and the internal state of the weight update algorithm.

32. The data generating method according to claim 9 or claim 31.

33. The semiconductor integrated circuit includes: Memory, A control processor; an accelerator; 33. A data generating method according to any one of claims 1 to 32.

34. A program for causing at least one computer to execute the data generating method according to any one of claims 1 to 33.

35. A data generating device for carrying out the data generating method according to any one of claims 1 to 33.

36. Using the data generated by the data generation method according to any one of claims 1 to 33, a calculation is performed according to a neural network trained using a deep learning framework. Semiconductor integrated circuit.

Images