Neural network generation method and neural network generation program

Abstract

To create a circuit and a model for performing an arithmetic operation related to a neural network which can be integrated in an integration device such as an IoT device, and can be operated efficiently and at a high speed.SOLUTION: A neural network creation method is a method for creating, from a neural network functional model, a neural network execution model which separately executes a convolution operation and a quantization operation, and includes a functional model creation step for creating a neural network functional model by using a single first operator which consecutively executes the convolution operation and the quantization operation to input data.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to a neural network generation method and a neural network generation program.

Background Art

[0002] In recent years, a convolutional neural network (CNN) has been used as a model for image recognition and the like. The convolutional neural network has a multi-layer structure including a convolutional layer and a pooling layer, and requires a large number of operations such as convolutional operations. Various operation methods for accelerating the operations by the convolutional neural network have been devised (such as Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, image recognition and the like using a convolutional neural network are also used in embedded devices such as IoT devices. In order to efficiently operate a convolutional neural network in an embedded device, it is desired to generate a circuit or a model that performs operations related to a neural network adapted to the hardware configuration of the embedded device.

[0005] In view of the above circumstances, an object of the present invention is to provide a neural network generation method and a neural network generation program that can be embedded in an embedded device such as an IoT device and can operate with high efficiency and high speed to generate a circuit or a model that performs operations related to a neural network.

Means for Solving the Problems

[0006] To solve the above problems, this invention proposes the following means. A neural network generation method according to a first aspect of the present invention is a method for generating a neural network execution model that separately performs convolution and quantization operations from a neural network functional model, comprising a functional model generation step of generating the neural network functional model using a single first operator that successively performs the convolution and quantization operations on input data.

[0007] A neural network generation program according to a second aspect of the present invention is a program that causes a computer to generate a neural network execution model that separately performs convolution and quantization operations from a neural network functional model, wherein the program causes the computer to generate the neural network functional model using a single first operator that successively performs the convolution and quantization operations on input data. [Effects of the Invention]

[0008] The neural network generation method and neural network generation program of the present invention can be embedded in embedded devices such as IoT devices and can generate neural networks that can operate with high performance. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows a neural network generation device according to the first embodiment. [Figure 2] This diagram shows the input and output of the processing unit of the neural network generator. [Figure 3] This figure shows an example of a convolutional neural network. [Figure 4] This diagram illustrates the convolution operation performed by the convolutional layer of the convolutional neural network. [Figure 5]This is a diagram showing an example of a neural network execution model. [Figure 6] This is a timing chart showing an operation example of the neural network execution model. [Figure 7] This is a control flowchart of the neural network generation device. [Figure 8] This is a diagram showing an example of a GUI image for setting an NN function model. [Figure 9] This is an internal block diagram of the generated convolutional operation circuit. [Figure 10] This is an internal block diagram of the multiplier of the convolutional operation circuit. [Figure 11] This is an internal block diagram of the product-sum operation unit of the multiplier. [Figure 12] This is an internal block diagram of the accumulator circuit of the convolutional operation circuit. [Figure 13] This is an internal block diagram of the accumulator unit of the accumulator circuit. [Figure 14] This is a state transition diagram of the control circuit of the convolutional operation circuit. [Figure 15] This is an internal block diagram of the generated quantization operation circuit. [Figure 16] This is an internal block diagram of the vector operation circuit and the quantization circuit of the quantization operation circuit. [Figure 17] This is a block diagram of the operation unit of the vector operation circuit. [Figure 18] This is an internal block diagram of the quantization unit of the quantization circuit. [Figure 19] This is an internal block diagram of the generated DMAC. [Figure 20] This is a diagram for explaining data division and data expansion of the convolutional operation. [Figure 21] This is a timing chart showing another allocation example to neural network hardware.

Embodiments for Carrying Out the Invention

[0010] (First Embodiment) The first embodiment of the present invention will be described with reference to FIGS. 1 to 21. FIG. 1 is a diagram showing a neural network generation device 300 according to the present embodiment.

[0011] [Neural Network Generation Device 300] The neural network generation device 300 is a device that generates a learned neural network execution model 100 that can be incorporated into embedded devices such as IoT devices. The neural network execution model 100 is a software or hardware model generated to cause a convolutional neural network 200 (hereinafter referred to as "CNN 200" or "NN function model 200"), which is a neural network function model, to perform calculations in an embedded device. The NN function model 200 is a function model corresponding to the neural network execution model 100 and has the same functions as the neural network execution model 100.

[0012] The neural network generation device 300 is a program-executable device (computer) equipped with hardware such as a processor such as a CPU (Central Processing Unit) and a memory. The functions of the neural network generation device 300 are realized by executing a neural network generation program and a software generation program in the neural network generation device 300. The neural network generation device 300 includes a storage unit 310, an arithmetic unit 320, a data input unit 330, a data output unit 340, a display unit 350, and an operation input unit 360.

[0013] The memory unit 310 stores hardware information HW, network information NW, training dataset DS, neural network execution model 100 (hereinafter referred to as "NN execution model 100"), and trained parameters PM. Hardware information HW, training dataset DS, and network information NW are input data that are input to the neural network generator 300. The NN execution model 100 and trained parameters PM are output data that the neural network generator 300 outputs. Note that "trained NN execution model 100" includes the NN execution model 100 and trained parameters PM.

[0014] Hardware information HW is information about the embedded device (hereinafter referred to as "operating hardware") on which the NN execution model 100 operates. Hardware information HW includes, for example, the device type, device constraints, memory configuration, bus configuration, operating frequency, power consumption, and manufacturing process type of the operating hardware. Device type is, for example, ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array). Device constraints are, for example, the upper limit on the number of arithmetic units included in the operating device and the upper limit on the circuit size. Memory configuration is, for example, memory type, number of memory modules, memory capacity, and input / output data width. Bus configuration is, for example, bus type, bus width, bus communication standard, and connected devices on the same bus. In addition, if there are multiple variations of the NN execution model 100, hardware information HW will include information about the variation of the NN execution model 100 to be used.

[0015] Network information NW is information about CNN200 (NN Functional Model 200). Network information NW includes, for example, information defining the functions of CNN200 (NN Functional Model 200). Network information NW includes, for example, the network configuration of CNN200, input data information, output data information, quantization information, etc. Input data information includes the input data type, such as images or audio, and the input data size, etc.

[0016] The training dataset DS includes training data D1 used for training and test data D2 used for inference testing.

[0017] Figure 2 shows the input and output of the arithmetic unit 320. The arithmetic unit 320 includes an execution model generation unit 321, a learning unit 322, an inference unit 323, a hardware generation unit 324, a software generation unit 325, and a functional model generation unit 326. The network information NW and NN execution model 100 input to the arithmetic unit 320 may be generated by a device other than the neural network generation device 300.

[0018] The execution model generation unit 321 generates an NN execution model 100 based on hardware information HW and network information NW. The NN execution model 100 is a software and hardware model generated to perform calculations on the target hardware using CNN200. The software includes software to control the hardware model. The hardware model may be at the behavior level, at the RTL (Register Transfer Level), at the netlist representing connections between gates and circuit modules, or a combination thereof.

[0019] The learning unit 322 generates trained parameters PM using the NN execution model 100 and training data D1. The inference unit 323 performs inference tests using the NN execution model 100 and test data D2.

[0020] The hardware generation unit 324 generates a neural network hardware model 400 based on the hardware information HW and the NN execution model 100. The neural network hardware model 400 is a hardware model that can be implemented on the target hardware. The neural network hardware model 400 is optimized for the target hardware based on the hardware information HW. The neural network hardware model 400 may be an RTL (Register Transfer Level), a netlist representing connections between gates and circuit modules, or a combination thereof. The neural network hardware model 400 may also be a parameter list or configuration file necessary for implementing the NN execution model 100 on hardware. The parameter list or configuration file is used in combination with the separately generated NN execution model 100.

[0021] In the following explanation, a system that implements the neural network hardware model 400 on target hardware will be referred to as "neural network hardware 600".

[0022] The software generation unit 325 generates software 500 to operate the neural network hardware 600 based on network information NW and the NN execution model 100. The software 500 includes software to transfer learned parameters PM to the neural network hardware 600 as needed.

[0023] The functional model generation unit 326 generates (configures) a CNN200 (NN functional model 200) based on input from the user and outputs network information NW, which is information related to the CNN200 (NN functional model 200).

[0024] The data input unit 330 receives hardware information HW and network information NW, etc., necessary for generating the trained NN execution model 100. The hardware information HW and network information NW, etc., are input as data written in a predetermined data format, for example. The input hardware information HW and network information NW, etc., are stored in the storage unit 310. The hardware information HW and network information NW, etc., may also be input or modified by the user via the operation input unit 360.

[0025] The generated trained NN execution model 100 is output to the data output unit 340. For example, the generated NN execution model 100 and the trained parameters PM are output to the data output unit 340.

[0026] The display unit 350 has a known monitor such as an LCD display. The display unit 350 can display a console screen for receiving GUI (Graphical User Interface) images and commands generated by the calculation unit 320. Furthermore, when the calculation unit 320 requires information input from the user, the display unit 350 can display a message prompting the user to input information from the operation input unit 360, or a GUI image necessary for information input.

[0027] The operation input unit 360 is a device that allows the user to input instructions to the calculation unit 320, etc. The operation input unit 360 is a known input device such as a touch panel, keyboard, or mouse. The input from the operation input unit 360 is transmitted to the calculation unit 320.

[0028] All or part of the functions of the arithmetic unit 320 are realized by one or more processors, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), executing a program stored in program memory. However, all or part of the functions of the arithmetic unit 320 may also be realized by hardware (e.g., circuitry) such as an LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or PLD (Programmable Logic Device). Furthermore, all or part of the functions of the arithmetic unit 320 may also be realized by a combination of software and hardware.

[0029] All or part of the functions of the arithmetic unit 320 may be implemented using external accelerators such as CPUs, GPUs, or hardware installed on external devices such as cloud servers. The arithmetic unit 320 can improve its calculation speed by, for example, using a high-performance GPU or dedicated hardware on a cloud server in combination with the arithmetic unit 320.

[0030] The memory unit 310 is implemented using flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read-Only Memory), RAM (Random Access Memory), etc. All or part of the memory unit 310 may be provided on an external device such as a cloud server and connected to the arithmetic unit 320 etc. via a communication line.

[0031] The neural network generation device 300 may be composed of multiple devices (computers), and the functional blocks of the arithmetic unit 320 may be distributed among multiple devices. For example, the neural network generation device 300 may be separated into a first device (computer) having a functional model generation unit 326, a second device (computer) having an execution model generation unit 321, a third device (computer) having a learning unit 322 and an inference unit 323, and a fourth device (computer) having a hardware generation unit 324 and a software generation unit 325.

[0032] [Convolutional Neural Network (CNN) 200] Next, we will explain CNN200. Figure 3 shows an example of CNN200. The network information NW of CNN200 is information about the configuration of CNN200, which will be explained below. CNN200 uses low-bit weights w and quantized input data a, making it easy to embed in embedded devices.

[0033] CNN200 is a multilayer network that includes a convolutional layer 210 for performing convolution operations, a quantization layer 220 for performing quantization operations, and an output layer 230. In at least a portion of CNN200, the convolutional layer 210 and the quantization layer 220 are alternately connected. CNN200 is a model widely used for image recognition and video recognition. CNN200 may further have layers with other functions, such as fully connected layers.

[0034] Figure 4 illustrates the convolution operation performed by the convolutional layer 210. The convolutional layer 210 performs a convolution operation on the input data a using the weights w. The convolutional layer 210 also performs a sum-of-products operation with the input data a and the weights w as inputs.

[0035] The input data a (also called activation data or feature map) to the convolutional layer 210 is multidimensional data such as image data. In this embodiment, the input data a is a 3D tensor consisting of elements (x, y, c). The convolutional layer 210 of the CNN200 performs a convolution operation on the low-bit input data a. In this embodiment, the elements of the input data a are 2-bit unsigned integers (0, 1, 2, 3). The elements of the input data a may also be, for example, 4-bit or 8-bit unsigned integers.

[0036] If the input data to CNN200 is in a different format from the input data a to the convolutional layer 210, such as a 32-bit floating-point type, CNN200 may have an additional input layer before the convolutional layer 210 to perform type conversion and quantization.

[0037] The weights w (also called filters or kernels) of the convolutional layer 210 are multidimensional data having elements that are learnable parameters. In this embodiment, the weights w are a 4-dimensional tensor consisting of elements (i, j, c, d). The weights w have d 3-dimensional tensors (hereinafter referred to as "weights w") consisting of elements (i, j, c). The weights w in a trained CNN200 are trained data. The convolutional layer 210 of the CNN200 performs convolution operations using low-bit weights w. In this embodiment, the elements of the weights w are 1-bit signed integers (0, 1), where the value "0" represents +1 and the value "1" represents -1.

[0038] The convolutional layer 210 performs the convolution operation shown in Equation 1 and outputs output data f. In Equation 1, s represents the stride. In Figure 4, the region shown by the dotted line represents one of the regions ao (hereinafter referred to as "application region ao") to which the weight wo is applied to the input data a. The elements of the application region ao are represented by (x+i, y+j, c).

[0039]

number

[0040] The quantization layer 220 performs quantization on the output of the convolution operation output by the convolution layer 210. The quantization layer 220 includes a pooling layer 221, a batch normalization layer 222, an activation function layer 223, and a quantization layer 224.

[0041] The pooling layer 221 compresses the output data f of the convolution operation output by the convolution layer 210 by performing operations such as average pooling (Equation 2) and MAX pooling (Equation 3). In Equations 2 and 3, u represents the input tensor, v represents the output tensor, and T represents the size of the pooling region. In Equation 3, max is a function that outputs the maximum value of u for the combination of i and j included in T.

[0042]

number

[0043]

number

[0044] The Batch Normalization layer 222 normalizes the data distribution of the output data from the quantization layer 220 and the pooling layer 221 using an operation such as that shown in Equation 4. In Equation 4, u represents the input tensor, v represents the output tensor, α represents the scale, and β represents the bias. In the trained CNN200, α and β are trained constant vectors.

[0045]

number

[0046] The activation function layer 223 performs activation function operations such as ReLU (Equation 5) on the outputs of the quantization operation layer 220, the pooling layer 221, and the Batch Normalization layer 222. In Equation 5, u is the input tensor and v is the output tensor. In Equation 5, max is the function that outputs the largest number among the arguments.

[0047]

number

[0048] The quantization layer 224 performs quantization on the outputs of the pooling layer 221 and the activation function layer 223 based on the quantization parameters, for example, as shown in Equation 6. The quantization shown in Equation 6 reduces the input tensor u to 2 bits. In Equation 6, q(c) is a vector of quantization parameters. In a trained CNN200, q(c) is a trained constant vector. The inequality sign "≦" in Equation 6 may also be "<".

[0049]

number

[0050] The output layer 230 is the layer that outputs the results of CNN200 using an identity function, a softmax function, or the like. The layer preceding the output layer 230 may be a convolutional layer 210 or a quantization layer 220.

[0051] In CNN200, the output data of the quantized quantization layer 224 is input to the convolutional layer 210, resulting in a lower computational load on the convolutional layer 210 compared to other convolutional neural networks that do not perform quantization.

[0052] [Neural Network Execution Model 100 (NN Execution Model) 100] Next, the NN execution model 100 will be described. Figure 5 shows an example of the NN execution model 100. The NN execution model 100 is software and hardware models generated to perform calculations on the target hardware using CNN200 (NN functional model 200). The software includes software that controls the hardware model, such as software that uses an instruction set to control the hardware model. The hardware model may be at the behavior level, at the RTL (Register Transfer Level), at the netlist representing connections between gates and circuit modules, or a combination of these.

[0053] The NN execution model 100 comprises a first memory 1, a second memory 2, a DMA controller 3 (hereinafter also referred to as "DMAC3"), a convolution operation circuit 4, a quantization operation circuit 5, and a controller 6. The NN execution model 100 is characterized in that the convolution operation circuit 4 and the quantization operation circuit 5 are formed in a loop via the first memory 1 and the second memory 2.

[0054] The first memory 1 is a rewritable memory, such as a volatile memory composed of SRAM (Static RAM). Data is written to and read from the first memory 1 via the DMAC 3 and controller 6. The first memory 1 is connected to the input port of the convolution arithmetic circuit 4, and the convolution arithmetic circuit 4 can read data from the first memory 1. The first memory 1 is also connected to the output port of the quantization arithmetic circuit 5, and the quantization arithmetic circuit 5 can write data to the first memory 1. The external host CPU can input and output data to and from the NN execution model 100 by writing and reading data to and from the first memory 1.

[0055] The second memory 2 is a rewritable memory, such as a volatile memory composed of SRAM (Static RAM). Data is written to and read from the second memory 2 via the DMAC 3 and controller 6. The second memory 2 is connected to the input port of the quantization arithmetic circuit 5, and the quantization arithmetic circuit 5 can read data from the second memory 2. The second memory 2 is also connected to the output port of the convolution arithmetic circuit 4, and the convolution arithmetic circuit 4 can write data to the second memory 2. The external host CPU can input and output data to and from the NN execution model 100 by writing to and reading data from the second memory 2.

[0056] DMAC3 is connected to the external bus EB and performs data transfer between the external memory, such as DRAM, and the first memory 1. DMAC3 also performs data transfer between the external memory, such as DRAM, and the second memory 2. Furthermore, DMAC3 performs data transfer between the external memory, such as DRAM, and the convolution arithmetic circuit 4. Finally, DMAC3 performs data transfer between the external memory, such as DRAM, and the quantization arithmetic circuit 5.

[0057] The convolution operation circuit 4 is a circuit that performs convolution operations in the convolution layer 210 of the trained CNN200. The convolution operation circuit 4 reads the input data a stored in the first memory 1 and performs a convolution operation on the input data a. The convolution operation circuit 4 writes the output data f of the convolution operation (hereinafter also referred to as "convolution operation output data") to the second memory 2.

[0058] The quantization circuit 5 is a circuit that performs at least a portion of the quantization operations in the quantization layer 220 of the trained CNN200. The quantization circuit 5 reads the output data f of the convolution operation stored in the second memory 2 and performs a quantization operation (an operation that includes at least quantization, among pooling, batch normalization, activation function, and quantization) on the output data f of the convolution operation. The quantization circuit 5 writes the output data of the quantization operation (hereinafter also referred to as "quantization operation output data") out to the first memory 1.

[0059] Controller 6 is connected to the external bus EB and operates as a slave to an external host CPU. Controller 6 has registers 61, which include parameter registers and state registers. The parameter registers control the operation of the NN execution model 100. The state registers indicate the state of the NN execution model 100, including the semaphore S. The external host CPU can access registers 61 via Controller 6.

[0060] Controller 6 is connected to the first memory 1, the second memory 2, the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via the internal bus IB. An external host CPU can access each block via Controller 6. For example, the external host CPU can issue instructions to the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via Controller 6. In addition, the DMAC 3, the convolution circuit 4, and the quantization circuit 5 can update the state registers (including semaphores S) of Controller 6 via the internal bus IB. The state registers (including semaphores S) may be configured to be updated via dedicated wiring connected to the DMAC 3, the convolution circuit 4, and the quantization circuit 5.

[0061] Since the NN execution model 100 has a first memory 1 and a second memory 2, etc., it can reduce the number of data transfers of duplicate data when transferring data from external memory such as DRAM by DMAC3. This can significantly reduce the power consumption generated by memory access.

[0062] Figure 6 is a timing chart showing an example of the operation of the NN execution model 100. The NN execution model 100 performs calculations for the CNN200, which has a multilayer structure of multiple layers, using a loop-shaped circuit. The NN execution model 100 can efficiently utilize hardware resources due to its loop-shaped circuit configuration. The operation example of the neural network hardware 600 shown in Figure 6 will be explained below.

[0063] DMAC3 stores the input data a from Layer 1 (see Figure 3) in the first memory 1. DMAC3 may also divide the input data a from Layer 1 and transfer it to the first memory 1 in accordance with the order of the convolution operations performed by the convolution operation circuit 4.

[0064] The convolution circuit 4 reads the input data a of Layer 1 (see Figure 3) stored in the first memory 1. The convolution circuit 4 performs a convolution operation on the input data a of Layer 1. The output data f of the convolution operation of Layer 1 is stored in the second memory 2.

[0065] The quantization circuit 5 reads the output data f of Layer 1 stored in the second memory 2. The quantization circuit 5 performs the Layer 2 quantization operation on the output data f of Layer 1. The output data out of the Layer 2 quantization operation is stored in the first memory 1.

[0066] The convolution circuit 4 reads the output data of the quantization operation of layer 2 stored in the first memory 1. The convolution circuit 4 uses the output data out of the quantization operation of layer 2 as input data a and performs the convolution operation of layer 3. The output data f of the convolution operation of layer 3 is stored in the second memory 2.

[0067] The convolution circuit 4 reads out the output data out from the quantization operation of layer 2M-2 (where M is a natural number) stored in the first memory 1. The convolution circuit 4 uses the output data out from the quantization operation of layer 2M-2 as input data a and performs the convolution operation of layer 2M-1. The output data f from the convolution operation of layer 2M-1 is stored in the second memory 2.

[0068] The quantization circuit 5 reads the output data f of layer 2M-1 stored in the second memory 2. The quantization circuit 5 performs a quantization operation on the output data f of layer 2M-1. The output data out of the quantization operation of layer 2M is stored in the first memory 1.

[0069] The convolution circuit 4 reads the output data out from the quantization operation of layer 2M stored in the first memory 1. The convolution circuit 4 uses the output data out from the quantization operation of layer 2M as input data a and performs a convolution operation on layer 2M+1. The output data f from the convolution operation of layer 2M+1 is stored in the second memory 2.

[0070] The convolution circuit 4 and the quantization circuit 5 perform operations alternately, advancing the CNN200 operation shown in Figure 3. In the NN execution model 100, the convolution circuit 4 performs the convolution operations on layers 2M-1 and 2M+1 using time-division multiplexing. In addition, the NN execution model 100 performs the quantization operations on layers 2M-2 and 2M using time-division multiplexing on the quantization circuit 5. Therefore, the NN execution model 100 has a significantly smaller circuit size compared to the case where separate convolution circuit 4 and quantization circuit 5 are implemented for each layer.

[0071] [Operation of the neural network generator 300] Next, the operation of the neural network generator 300 (neural network control method) will be explained according to the control flowchart of the neural network generator 300 shown in Figure 7. After performing an initialization process, the neural network generator 300 executes step S10.

[0072] <Neural network functional model generation process (S10)> In step S10, the functional model generation unit 326 of the neural network generation device 300 generates a CNN200 (NN functional model 200) and outputs network information NW, which is information related to the CNN200 (NN functional model 200) (neural network functional model generation process). For example, the functional model generation unit 326 generates the CNN200 (NN functional model 200) by displaying a GUI image to be set up for the CNN200 (NN functional model 200) on the display unit 350 and having the user input the necessary information from the operation input unit 360.

[0073] The functional model generation unit 326 may include libraries or platforms (e.g., TensorFlow or PyTorch) that can be used to generate functional models of known neural networks.

[0074] Figure 8 shows an example of a GUI image for configuring the NN function model 200. The functional model generation unit 326 sets the network structure and layer-specific specifications for the CNN200 (NN functional model 200) based on user input from the operation input unit 360. For example, the user changes the network structure of the NN functional model 200 by rearranging the connections of the layers, which are visually represented as a GUI image. The user also changes the layer-specific specifications (input data information, output data information, quantization information, etc.) of the layers, which are visually represented as a GUI image.

[0075] The layers of the NN functional model 200 set by the functional model generation unit 326 are not limited to the convolutional layer 210, quantization layer 220, and output layer 230 shown in Figure 3. The functional model generation unit 326 can set a "functional layer," which combines multiple functions of a neural network into a single layer, as part of the NN functional model 200. In addition, the functional model generation unit 326 can set a "verification layer," which includes a function to evaluate the performance of the NN execution model 100, as part of the NN functional model 200.

[0076] The first operator (first function) OP1 is one of the "function layers" and is a single operator (function) that performs convolution and quantization operations in succession. The first operator OP1 performs a convolution operation using weights w on the quantized input data a, performs a quantization operation on the convolved data, and outputs the quantized data. By providing combinations of operations that are frequently used in the NN function model 200 as "function layers," the workload on the user when setting up the NN function model 200 can be reduced. Here, the first operator OP1 may also be an operator that takes modifiable operation parameters such as the number of input / output channels, filter size, and stride as inputs. Furthermore, the first operator OP1 may also be an operator that takes the quantization parameters of the quantization operation as inputs. The quantization parameters may include quantization parameters for output data and quantization parameters for weights. Specifically, the quantization parameters (function / threshold) for output data are read in the activation operation and used in the quantization operation. The quantization parameters (function / threshold) for weights are read and used in the convolution operation. Note that the weight w, which is one of the inputs to the first operator OP1, may be quantized at the time of input.

[0077] The first operator (first function) OP1 may be a single operator (function) that performs Batch Normalization in addition to the convolution and quantization operations.

[0078] The second operator (second function) OP2 is one of the "verification layers" and is a single operator (function) that performs a convolution operation. The second operator OP2 performs a convolution operation on the input data using weights w and outputs the output data of the convolution operation. The second operator OP2 can quantize at least a portion of the convolution operation. When quantization of the convolution operation is disabled, the second operator (second function) OP2 performs a convolution operation on the input data using weights w and outputs the output data of the convolution operation as is. The convolution operation is performed using high-precision calculations (e.g., floating-point calculations). On the other hand, when quantization of the convolution operation is enabled, the input data, weights w, and at least a portion of the output data are quantized. For example, to match the data input to the convolution operation circuit 4 of the neural network execution model 100, the input data is quantized to 2 bits and the weights w are quantized to 1 bit. Furthermore, the output data is quantized to 2 bits to match the quantization data output from the quantization circuit 5 of the neural network execution model 100. In other words, by switching whether or not the convolution operation is quantized in OP2, it is possible to easily compare the NN function model 200 that performs the convolution operation with high precision with the NN function model 200 that performs the convolution operation with the precision of the neural network execution model 100. As an example, it is preferable to use the second operator (second function) OP2 in the output layer 230 or a layer close to the output layer 230 of the NN function model 200. This makes it easy to grasp the effect of quantizing at least a part of the convolution operation in the output layer 230.

[0079] Furthermore, the definition of the functions of CNN200 (NN Functional Model 200) does not necessarily have to be described in a visually diagrammatic manner as illustrated in Figure 8. The definition of the functions of CNN200 (NN Functional Model 200) may be described, for example, in a programming language or XML.

[0080] Furthermore, the GUI image used to configure the NN functional model 200 may also display design support information to reduce the workload for the user when configuring the NN functional model 200. For example, the displayed design support information may indicate whether the configured NN functional model 200 is a functional model capable of generating the neural network execution model 100. Specifically, the design support information may include information indicating whether the quantization function used in the NN functional model 200 is a function that can be accelerated by the neural network execution model 100, information indicating whether the input data and weights of each layer are in an appropriate format, information indicating whether the size (capacity) of various parameters (weights, quantization parameters, etc.) is below a predetermined amount, and information indicating whether the number of channels in the input data is below a predetermined number.

[0081] Furthermore, the displayed design support information may also include information indicating whether the configured NN functional model 200 is a functional model capable of generating software 500. Specifically, the design support information may include information indicating whether the quantization function used in the NN functional model 200 is a function that can be converted into an executable instruction command C5 in the neural network execution model 100.

[0082] <Hardware information acquisition process (S11)> In step S11, the neural network generator 300 acquires hardware information HW of the target hardware (hardware information acquisition step). The neural network generator 300 acquires, for example, the hardware information HW input to the data input unit 330. The neural network generator 300 may also acquire the hardware information HW by displaying the GUI image necessary for inputting the hardware information HW on the display unit 350 and having the user input the hardware information HW from the operation input unit 360.

[0083] Hardware information HW specifically includes the memory type, memory capacity, and input / output data width of the memory allocated as the first memory 1 and the second memory 2.

[0084] The acquired hardware information HW is stored in the memory unit 310. Next, the neural network generator 300 performs step S12.

[0085] <Network information acquisition process (S12)> In step S12, the neural network generation device 300 acquires the network information NW of the CNN200 generated in the neural network generation process (S10) (network information acquisition process).

[0086] The acquired network information NW is stored in the storage unit 310. Next, the neural network generator 300 performs step S13.

[0087] <Neural network execution model generation process (S13)> In step S13, the execution model generation unit 321 of the neural network generation device 300 generates an NN execution model 100 based on hardware information HW and network information NW (neural network execution model generation step).

[0088] The neural network execution model generation process (NN execution model generation process) includes, for example, a convolution operation circuit generation process (S13-1), a quantization operation circuit generation process (S13-2), and a DMAC generation process (S13-3).

[0089] <Convolution arithmetic circuit generation process (S13-1)> The execution model generation unit 321 generates a convolution circuit 4 of the NN execution model 100 based on hardware information HW and network information NW (convolution circuit generation step). The execution model generation unit 321 generates a hardware model of the convolution circuit 4 from information such as weights w and the bit width of input data a, which are input as network information NW. The hardware model may be at the behavior level, at the RTL (Register Transfer Level), at the netlist representing connections between gates and circuit modules, or a combination thereof. An example of the generated hardware model of the convolution circuit 4 is described below.

[0090] Figure 9 is an internal block diagram of the generated convolution operation circuit 4. The convolution arithmetic circuit 4 includes a weight memory 41, a multiplier 42, an accumulator circuit 43, and a state controller 44. The convolution arithmetic circuit 4 has a dedicated state controller 44 for the multiplier 42 and the accumulator circuit 43, and when an instruction command is input, it can perform a convolution operation without requiring an external controller.

[0091] The weight memory 41 is a memory that stores the weights w used in the convolution operation, and is a rewritable memory such as a volatile memory composed of SRAM (Static RAM), for example. The DMAC3 writes the weights w required for the convolution operation to the weight memory 41 via DMA transfer.

[0092] Figure 10 is an internal block diagram of the multiplier 42. The multiplier 42 multiplies each element of the input vector a by each element of the weight matrix w. Each element of the input vector a is a vector data with Bc elements, obtained by dividing the input data a (for example, "input vector A" described later). Similarly, each element of the weight matrix w is a matrix data with Bc × Bd elements, obtained by dividing the weight w (for example, "weight matrix W" described later). The multiplier 42 has Bc × Bd multiply-accumulate units 47, and can perform the multiplication of the input vector A and the weight matrix W in parallel.

[0093] The multiplier 42 reads the input vector A and weight matrix W required for multiplication from the first memory 1 and weight memory 41, and performs the multiplication. The multiplier 42 outputs the result of Bd sum-of-accumulate operations O(di).

[0094] Figure 11 is an internal block diagram of the sum-of-accumulate unit 47. The multiply-accumulate unit 47 multiplies the input vector A's element A(ci) by the weight matrix W's element W(ci,di). The multiply-accumulate unit 47 then adds the result of the multiplication to the result S(ci,di) of another multiply-accumulate unit 47. The multiply-accumulate unit 47 outputs the summation result S(ci+1,di). ci is an index from 0 to (Bc-1). di is an index from 0 to (Bd-1). Element A(ci) is a 2-bit unsigned integer (0,1,2,3). Element W(ci,di) is a 1-bit signed integer (0,1), where the value "0" represents +1 and the value "1" represents -1.

[0095] The multiply-accumulate unit 47 includes an inverter 47a, a selector 47b, and an adder 47c. The multiply-accumulate unit 47 performs multiplication using only the inverter 47a and selector 47b, without using a multiplier. Selector 47b selects the input of element A(ci) when element W(ci,di) is "0". Selector 47b selects the complement of element A(ci) inverted by the inverter when element W(ci,di) is "1". Element W(ci,di) is also input to the carry-in of adder 47c. Adder 47c outputs the value obtained by adding element A(ci) to S(ci,di) when element W(ci,di) is "0". Adder 47c outputs the value obtained by subtracting element A(ci) from S(ci,di) when W(ci,di) is "1".

[0096] Figure 12 is an internal block diagram of the accumulator circuit 43. The accumulator circuit 43 accumulates the sum-of-accumulate result O(di) of the multiplier 42 in the second memory 2. The accumulator circuit 43 has Bd accumulator units 48 and can accumulate Bd sum-of-accumulate results O(di) in the second memory 2 in parallel.

[0097] Figure 13 is an internal block diagram of the accumulator unit 48. The accumulator unit 48 includes an adder 48a and a mask unit 48b. The adder 48a adds the elements O(di) of the sum-of-accumulate result O to the partial sum, which is an intermediate result of the convolution operation shown in Equation 1 and stored in the second memory 2. The addition result is 16 bits per element. The addition result is not limited to 16 bits per element, but may be, for example, 15 bits or 17 bits per element.

[0098] The adder 48a writes the addition result to the same address in the second memory 2. The mask unit 48b, when the initialization signal clear is asserted, masks the output from the second memory 2 and sets the element O(di) to zero for addition. The initialization signal clear is asserted when the partial sum of the intermediate results is not stored in the second memory 2.

[0099] Once the convolution operation by the multiplier 42 and the accumulator circuit 43 is completed, output data f(x,y,do) with Bd elements is stored in the second memory.

[0100] The state controller 44 controls the states of the multiplier 42 and the accumulator circuit 43. The state controller 44 is also connected to the controller 6 via the internal bus IB. The state controller 44 has an instruction queue 45 and a control circuit 46.

[0101] The instruction queue 45 is a queue that stores instruction commands C4 for the convolution arithmetic circuit 4, and is composed of, for example, a FIFO memory. Instruction commands C4 are written to the instruction queue 45 via the internal bus IB.

[0102] The control circuit 46 is a state machine that decodes instruction command C4 and controls the multiplier 42 and accumulator circuit 43 based on instruction command C4. The control circuit 46 may be implemented by logic circuits or by a CPU controlled by software.

[0103] Figure 14 is a state transition diagram of the control circuit 46. When an instruction command C4 is input to the instruction queue 45 (not empty), the control circuit 46 transitions from the idle state S1 to the decode state S2.

[0104] In the decode state S2, the control circuit 46 decodes the instruction command C3 output from the instruction queue 45. The control circuit 46 also reads the semaphore S stored in the controller 6's register 61 and determines whether the operation of the multiplier 42 and accumulator circuit 43 instructed by the instruction command C4 is possible. If it is not possible (Not ready), the control circuit 46 waits until it becomes possible (Wait). If it is possible (Ready), the control circuit 46 transitions from the decode state S2 to the execution state S3.

[0105] In the execution state S3, the control circuit 46 controls the multiplier 42 and the accumulator circuit 43 to perform the operations instructed by the instruction command C4. When the operation of the multiplier 42 and the accumulator circuit 43 is complete, the control circuit 46 removes the completed instruction command C4 from the instruction queue 45 and updates the semaphore S stored in the controller 6's register 61. If there are instructions in the instruction queue 45 (Not empty), the control circuit 46 transitions from the execution state S3 to the decode state S2. If there are no instructions in the instruction queue 45 (empty), the control circuit 46 transitions from the execution state S3 to the idle state S1.

[0106] The execution model generation unit 321 determines the specifications and size (Bc and Bd) of the arithmetic units in the convolutional arithmetic circuit 4 from information such as the weights w and the bit width of the input data a, which are input as network information NW. If the hardware information HW includes the hardware size of the NN execution model 100 (neural network hardware model 400, neural network hardware 600) to be generated, the execution model generation unit 321 adjusts the specifications and size (Bc and Bd) of the arithmetic units in the convolutional arithmetic circuit 4 to match the specified size.

[0107] <Quantization operation circuit generation process (S13-2)> The execution model generation unit 321 generates a quantization arithmetic circuit 5 of the NN execution model 100 based on hardware information HW and network information NW (quantization arithmetic circuit generation step). The execution model generation unit 321 generates a hardware model of the quantization arithmetic circuit 5 from the quantization information input as network information NW. The hardware model may be at the behavior level, at the RTL (Register Transfer Level), at the netlist representing connections between gates and circuit modules, or a combination thereof. An example of the generated hardware model of the quantization arithmetic circuit 5 is described below.

[0108] Figure 15 is an internal block diagram of the generated quantization operation circuit 5. The quantization operation circuit 5 includes a quantization parameter memory 51, a vector operation circuit 52, a quantization circuit 53, and a state controller 54. The quantization operation circuit 5 has a dedicated state controller 54 for the vector operation circuit 52 and the quantization circuit 53, and when an instruction command is input, it can perform quantization operations without requiring an external controller.

[0109] The quantization parameter memory 51 is a memory that stores the quantization parameters q used in quantization calculations, and is a rewritable memory such as a volatile memory composed of SRAM (Static RAM), for example. The DMAC3 writes the quantization parameters q required for quantization calculations to the quantization parameter memory 51 by DMA transfer.

[0110] Figure 16 is an internal block diagram of the vector operation circuit 52 and the quantization circuit 53. The vector arithmetic circuit 52 performs operations on the output data f(x,y,do) stored in the second memory 2. The vector arithmetic circuit 52 has Bd arithmetic units 57 and performs SIMD operations in parallel on the output data f(x,y,do).

[0111] Figure 17 is a block diagram of the arithmetic unit 57. The arithmetic unit 57 includes, for example, an ALU 57a, a first selector 57b, a second selector 57c, a register 57d, and a shifter 57e. The arithmetic unit 57 may further include other arithmetic units, etc., that are present in known general-purpose SIMD arithmetic circuits.

[0112] The vector operation circuit 52 combines the arithmetic units and other components of the operation unit 57 to perform at least one operation on the output data f(x,y,do) from among the operations of the pooling layer 221, the batch normalization layer 222, and the activation function layer 223 in the quantization operation layer 220.

[0113] The arithmetic unit 57 can add the data stored in register 57d and the element f(di) of the output data f(x,y,do) read from the second memory 2 using ALU 57a. The arithmetic unit 57 can store the addition result from ALU 57a in register 57d. The arithmetic unit 57 can initialize the addition result by inputting "0" to ALU 57a in place of the data stored in register 57d, depending on the selection of the first selector 57b. For example, if the pooling area is 2x2, the shifter 57e can output the average value of the addition result by right-shifting the output of ALU 57a by 2 bits. The vector arithmetic circuit 52 can perform the average pooling calculation shown in Equation 2 by repeating the above calculations by Bd arithmetic units 57.

[0114] The arithmetic unit 57 can compare the data stored in register 57d with the element f(di) of the output data f(x,y,do) read from the second memory 2 using the ALU 57a. The arithmetic unit 57 controls the second selector 57c according to the comparison result by ALU 57a to select the larger of the data stored in register 57d and element f(di). The arithmetic unit 57 can initialize the comparison target to the minimum value by inputting the minimum possible value of element f(di) to ALU 57a based on the selection of the first selector 57b. In this embodiment, since element f(di) is a 16-bit signed integer, the minimum possible value of element f(di) is "0x8000". The vector arithmetic circuit 52 can perform the MAX pooling calculation of equation 3 by repeating the above calculations by Bd arithmetic units 57. Note that in the MAX pooling calculation, the shifter 57e does not shift the output of the second selector 57c.

[0115] The arithmetic unit 57 can subtract the data stored in register 57d from the element f(di) of the output data f(x,y,do) read from the second memory 2 using ALU 57a. The shifter 57e can left-shift (i.e. multiply) or right-shift (i.e. divide) the output of ALU 57a. The vector arithmetic circuit 52 can perform the Batch Normalization operation of Equation 4 by repeating the above operations by Bd arithmetic units 57.

[0116] The arithmetic unit 57 can compare the element f(di) of the output data f(x,y,do) read from the second memory 2 with "0" selected by the first selector 57b using the ALU 57a. The arithmetic unit 57 can select and output either the element f(di) or the constant value "0" previously stored in register 57d, depending on the comparison result by the ALU 57a. The vector arithmetic circuit 52 can perform the ReLU operation of equation 5 by repeating the above operations by Bd arithmetic units 57.

[0117] The vector operation circuit 52 can perform mean pooling, MAX pooling, batch normalization, activation function calculations, and combinations thereof. Since the vector operation circuit 52 can perform general-purpose SIMD operations, it may also perform other operations necessary for the operations in the quantization operation layer 220. Furthermore, the vector operation circuit 52 may also perform operations other than those in the quantization operation layer 220.

[0118] Note that the quantization circuit 5 does not necessarily have a vector operation circuit 52. If the quantization circuit 5 does not have a vector operation circuit 52, the output data f(x,y,do) is input to the quantization circuit 53.

[0119] The quantization circuit 53 performs quantization on the output data of the vector operation circuit 52. As shown in Figure 16, the quantization circuit 53 has Bd quantization units 58 and performs operations in parallel on the output data of the vector operation circuit 52.

[0120] FIG. 18 is an internal block diagram of the quantization unit 58. The quantization unit 58 performs quantization on the elements in(di) of the output data of the vector operation circuit 52. The quantization unit 58 includes a comparator 58a and an encoder 58b. The quantization unit 58 performs the operation (Equation 6) of the quantization layer 224 in the quantization operation layer 220 on the output data (16 bits / element) of the vector operation circuit 52. The quantization unit 58 reads the necessary quantization parameters q(th0, th1, th2) from the quantization parameter memory 51, and the comparator 58a compares the input in(di) with the quantization parameters q. The quantization unit 58 quantizes the comparison result by the comparator 58a to 2 bits / element using the encoder 58b. Since α(c) and β(c) in Equation 4 are different parameters for each variable c, the quantization parameters q(th0, th1, th2) reflecting α(c) and β(c) are different parameters for each in(di).

[0121] The quantization unit 58 classifies the input in(di) into four regions (for example, in≦th0, th0<in≦th1, th1<in≦th2, th2<in) by comparing the input in(di) with three threshold values th0, th1, th2, encodes the classification result into 2 bits, and outputs it. The quantization unit 58 can also perform operations of Batch Normalization and activation functions in combination with quantization by setting the quantization parameters q(th0, th1, th2).

[0122] The quantization unit 58 can perform the operation of Batch Normalization shown in Equation 4 in combination with quantization by setting the threshold value th0 as β(c) in Equation 4 and the threshold differences (th1 - th0) and (th2 - th1) as α(c) in Equation 4. By increasing (th1 - th0) and (th2 - th1), α(c) can be decreased. By decreasing (th1 - th0) and (th2 - th1), α(c) can be increased.

[0123] The quantization unit 58 can perform the ReLU operation of the activation function in addition to the quantization of the input in(di). For example, the quantization unit 58 saturates the output value in the regions where in(di) ≤ th0 and th2 < in(di). The quantization unit 58 can perform the operation of the activation function in combination with quantization by setting the quantization parameter q so that the output becomes non-linear.

[0124] The state controller 54 controls the states of the vector operation circuit 52 and the quantization circuit 53. Also, the state controller 54 is connected to the controller 6 via the internal bus IB. The state controller 54 has an instruction queue 55 and a control circuit 56.

[0125] The instruction queue 55 is a queue in which the instruction command C5 for the quantization operation circuit 5 is stored, and is composed of, for example, a FIFO memory. The instruction command C5 is written into the instruction queue 55 via the internal bus IB.

[0126] The control circuit 56 is a state machine that decodes the instruction command C5 and controls the vector operation circuit 52 and the quantization circuit 53 based on the instruction command C5. The control circuit 56 has the same configuration as the control circuit 46 of the state controller 44 of the convolution operation circuit 4.

[0127] The quantization operation circuit 5 writes quantization operation output data having Bd elements into the first memory 1. A preferred relationship between Bd and Bc is shown in Equation 7. In Equation 7, n is an integer.

[0128] [Number]

[0129] The execution model generation unit 321 determines, from the quantization information input as the network information NW, the presence and type of pooling operations (such as average pooling, MAX pooling, etc.), the presence and method of Batch Normalization operations, the presence and method of activation function operations (such as ReLU operations), the quantization method (number of bits, etc.), and the presence of other operations in the quantization operation circuit 5. When the hardware scale of the NN execution model 100 (neural network hardware model 400, neural network hardware 600) to be generated is included in the hardware information HW, the execution model generation unit 321 adjusts the configuration of the arithmetic unit in the quantization operation circuit 5 according to the specified scale.

[0130] <DMAC Generation Step (S13-3)> The execution model generation unit 321 generates the DMAC 3 of the NN execution model 100 based on the hardware information HW and the network information NW (DMAC generation step). The execution model generation unit 321 generates the hardware model of the DMAC 3 from the information input as the network information NW. The hardware model may be at the behavioral level, may be RTL (Register Transfer Level), may be a netlist representing the connection between gates and circuit modules, or may be a combination thereof. Hereinafter, an example of the hardware model of the generated DMAC 3 will be described.

[0131] FIG. 19 is an internal block diagram of the generated DMAC 3. The DMAC 3 includes a data transfer circuit 31 and a state controller 32. The DMAC 3 has a dedicated state controller 32 for the data transfer circuit 31, and can perform DMA data transfer without requiring an external controller when an instruction command is input.

[0132] The data transfer circuit 31 is connected to the external bus EB and performs DMA data transfer between the external memory, such as DRAM, and the first memory 1. The data transfer circuit 31 also performs DMA data transfer between the external memory, such as DRAM, and the second memory 2. Furthermore, the data transfer circuit 31 performs data transfer between the external memory, such as DRAM, and the convolution arithmetic circuit 4. Finally, the data transfer circuit 31 performs data transfer between the external memory, such as DRAM, and the quantization arithmetic circuit 5. The number of DMA channels in the data transfer circuit 31 is not limited. For example, the first memory 1 and the second memory 2 may each have their own dedicated DMA channel.

[0133] The state controller 32 controls the state of the data transfer circuit 31. The state controller 32 is also connected to the controller 6 via the internal bus IB. The state controller 32 includes an instruction queue 33 and a control circuit 34.

[0134] The instruction queue 33 is a queue where instruction commands C3 for DMAC3 are stored, and is composed of, for example, a FIFO memory. One or more instruction commands C3 are written to the instruction queue 33 via the internal bus IB.

[0135] The control circuit 34 is a state machine that decodes the instruction command C3 and sequentially controls the data transfer circuit 31 based on the instruction command C3. The control circuit 34 has a similar configuration to the control circuit 46 of the state controller 44 of the convolution arithmetic circuit 4.

[0136] The execution model generation unit 321 determines the number of DMA channels, data bus width, and other parameters in the DMAC3 from the information input as network information NW.

[0137] For example, the execution model generation unit 321 generates a DMAC3 with specifications (data bus width, etc.) that match the specifications of the host-side external bus EB. By increasing the data bus width and the number of DMA channels, the data transmission speed between the external memory and the first memory 1 or second memory 2 can be improved.

[0138] <Learning Process (S14)> In step S14, the learning unit 322 and inference unit 323 of the neural network generator 300 learn the learning parameters of the generated NN execution model 100 using the learning dataset DS (learning process). The learning process (S14) includes, for example, a learned parameter generation process (S14-1) and an inference test process (S14-2).

[0139] <Learning Process: Process for Generating Learned Parameters (S14-1)> The learning unit 322 generates trained parameters PM using the NN execution model 100 and training data D1. The trained parameters PM are the trained weights w and quantization parameters q, etc.

[0140] For example, if the NN execution model 100 is an execution model of CNN200 that performs image recognition, then the training data D1 is a combination of input images and training data T. The input images are input data a that is input to CNN200. The training data T includes the type of subject captured in the image, the presence or absence of objects to be detected in the image, and the coordinate values ​​of objects to be detected in the image.

[0141] The learning unit 322 generates trained parameters PM using supervised learning, such as the backpropagation method, which is a well-known technique. The learning unit 322 calculates the difference E between the output of the NN execution model 100 for the input image and the training data T corresponding to the input image using a loss function (error function), and updates the weights w and quantization parameters q so that the difference E is minimized.

[0142] For example, when updating a weight w, the gradient of the loss function with respect to the weight w is used. The gradient is calculated, for example, by differentiating the loss function. When using backpropagation, the gradient is calculated by backpropagation.

[0143] The learning unit 322 improves the accuracy of operations related to convolution when calculating gradients and updating weights w. Specifically, 32-bit floating-point weights w, which are more accurate than the low-bit weights w (e.g., 1 bit) used by the NN execution model 100, are used for learning. In addition, the convolution operation performed in the convolution circuit 4 of the NN execution model 100 is made more accurate.

[0144] The learning unit 322 improves the accuracy of operations related to the activation function when calculating gradients and updating weights w. Specifically, a sigmond function, which is more accurate than activation functions such as the ReLU function performed in the quantization circuit 5 of the NN execution model 100, is used for learning.

[0145] On the other hand, when the learning unit 322 calculates output data for the input image by forward propagation, it does not perform calculations related to convolution and activation functions with high precision, but instead performs calculations based on the NN execution model 100. The high-precision weights w used when updating the weights w are reduced in bit depth using a lookup table or the like.

[0146] The learning unit 322 can generate trained parameter PMs that achieve high inference accuracy by improving the precision of operations related to convolution and activation functions when calculating gradients and updating weights w, thereby preventing a decrease in the precision of intermediate data in the operations.

[0147] On the other hand, when the learning unit 322 calculates output data for an input image, it does not perform forward propagation calculations with high precision, but instead performs calculations based on the NN execution model 100. Therefore, the output data calculated by the learning unit 322 matches the output data of the NN execution model 100 using the generated trained parameters PM.

[0148] When calculating output data for an input image by forward propagation, the learning unit 322 performs quantization in the convolution operation as defined in the second operator OP2 when the NN execution model 100 performs quantization in the convolution operation with quantization enabled in the second operator OP2. On the other hand, when the learning unit 322 performs quantization in the convolution operation with quantization disabled in the second operator OP2 in the NN execution model 100, it does not perform quantization in the convolution operation as defined in the second operator OP2. The user can easily compare the NN execution model 100 that performs quantization in the convolution operation at the corresponding layer with the NN execution model 100 that does not perform quantization at the corresponding layer, simply by switching whether or not quantization is performed in the second operator OP2 in the NN execution model 100 during the learning process.

[0149] <Learning Process: Inference Testing Process (S14-2)> The inference unit 323 performs an inference test using the trained parameters PM, the NN execution model 100, and the test data D2 generated by the learning unit 322. For example, if the NN execution model 100 is an execution model of CNN200 that performs image recognition, the test data D2 is a combination of input images and training data T, similar to the training data D1.

[0150] The inference unit 323 displays the progress and results of the inference test on the display unit 350. The results of the inference test are, for example, the accuracy rate for test data D2.

[0151] <Confirmation process (S15)> In step S15, the inference unit 323 of the neural network generator 300 displays a message prompting the user to input confirmation regarding the results from the operation input unit 360, as well as GUI images necessary for information input, on the display unit 350. The user inputs from the operation input unit 360 whether they accept the results of the inference test. If the user inputs from the operation input unit 360 indicating that they accept the results of the inference test, the neural network generator 300 then performs step S16. If the user inputs from the operation input unit 360 indicating that they do not accept the results of the inference test, the neural network generator 300 performs step S10 again to regenerate the NN function model 200 and re-outputs the network information NW (neural network function model regeneration process). In step S10, which is performed again, the user can change, for example, quantization information (such as whether or not quantization is performed for each layer) or input data information (such as the number of channels). Alternatively, the neural network generator 300 may return to step S11 without regenerating the NN function model 200 (network information NW) and have the user re-input the hardware information HW.

[0152] <Hardware manufacturing process (S16)> In step S16, the hardware generation unit 324 of the neural network generation device 300 generates a neural network hardware model 400 based on hardware information HW and the NN execution model 100 (hardware generation step).

[0153] <Software generation process (S17)> In step S17, the software generation unit 325 of the neural network generation device 300 generates software 500 that operates the neural network hardware 600 (a neural network hardware model 400 implemented on the target hardware) based on the NN function model 200 (network information NW) and the NN execution model 100. The software 500 is, for example, software that uses an instruction set to control the hardware model. The software 500 also includes software that transfers learned parameters PM to the neural network hardware 600 as needed.

[0154] The software generation process (S17) includes, for example, an input data splitting process (S17-1) and an allocation process (S17-2).

[0155] <Input data splitting process (S17-1): Data splitting> The software generation unit 325 divides the input data a for the convolution operation of the convolution layer 210 into subtensors based on the memory capacity of the memory allocated as the first memory 1 and the second memory 2, as well as the specifications and size (Bc and Bd) of the arithmetic unit. The method of dividing into subtensors and the number of divisions are not particularly limited. A subtensor is formed, for example, by dividing the input data a(x+i,y+j,c) into a(x+i,y+j,co).

[0156] Figure 20 illustrates the data partitioning and data expansion processes involved in convolution operations. In the data partitioning of the convolution operation, the variable c in Equation 1 is partitioned into blocks of size Bc, as shown in Equation 8. Similarly, the variable d in Equation 1 is partitioned into blocks of size Bd, as shown in Equation 9. In Equation 8, co is the offset and ci is the index from 0 to (Bc-1). In Equation 9, do is the offset and di is the index from 0 to (Bd-1). Note that sizes Bc and Bd may be the same.

[0157]

number

[0158]

number

[0159] In Equation 1, the input data a(x+i, y+j, c) is divided along the c-axis by size Bc, and the divided input data is represented as a(x+i, y+j, co). In the following explanation, the divided input data a will also be referred to as "divided input data a".

[0160] The weight w(i,j,c,d) in Equation 1 is divided by size Bc in the c-axis direction and size Bd in the d-axis direction, and is represented by the divided weight w(i,j,co,do). In the following explanation, the divided weight w will also be called the "divided weight w".

[0161] The output data f(x,y,do) divided by size Bd can be obtained using equation 10. By combining the divided output data f(x,y,do), the final output data f(x,y,d) can be calculated.

[0162]

number

[0163] <Input data splitting process (S17-1): Data expansion> The software generation unit 325 expands the divided input data a and weights w into the convolution circuit 4 of the NN execution model 100.

[0164] The segmented input data a(x+i, y+j, co) is expanded into vector data with Bc elements. The elements of the segmented input data a are indexed by ci (0 ≤ ci < Bc). In the following description, the segmented input data a expanded into vector data for each i and j is also referred to as "input vector A". The input vector A has elements from the segmented input data a(x+i, y+j, co×Bc) to the segmented input data a(x+i, y+j, co×Bc+(Bc-1)).

[0165] The segmented weight w(i, j, co, do) is expanded into matrix data with Bc×Bd elements. The elements of the segmented weight w expanded into matrix data are indexed by ci and di (0 ≤ di < Bd). In the following description, the segmented weight w expanded into matrix data for each i and j is also referred to as "weight matrix W". The weight matrix W has elements from the segmented weight w(i, j, co×Bc, do×Bd) to the segmented weight w(i, j, co×Bc+(Bc-1), do×Bd+(Bd-1)).

[0166] By multiplying the input vector A and the weight matrix W, vector data is calculated. By shaping the vector data calculated for each i, j, and co into a three-dimensional tensor, the output data f(x, y, do) can be obtained. By performing such data expansion, the convolution operation of the convolution layer 210 can be implemented by multiplying vector data and matrix data.

[0167] <Allocation Step (S17-2)> The software generation unit 325 generates software 500 that allocates the segmented operations to the neural network hardware 600 for execution (allocation step). The generated software 500 includes instruction commands C3, instruction command C4, and instruction command C5.

[0168] Software 500 causes the neural network execution model 100 to separately execute the convolution and quantization operations included in the first operator OP1 of the NN function model 200 (network information NW). Specifically, the convolution operation included in the first operator OP1 is converted to instruction command C4, and the quantization operation included in the first operator OP1 is converted to instruction command C5.

[0169] In the NN functional model 200, the convolution operation and the convolution operator (convolution function) to be executed, and the quantization operator (quantization function) to be executed, can be freely set by the user within a predetermined range, and therefore may not be convertible into executable instruction commands (C4, C5) in the neural network execution model 100.

[0170] On the other hand, in the NN functional model 200, the single first operator OP1, which executes the convolution operation and the quantization operation consecutively, is easy to separate and convert into executable instruction commands (C4, C5) in the neural network execution model 100 because the order of the convolution operation and the quantization operation are predetermined.

[0171] Figure 21 is a timing chart showing an example of allocation to the neural network hardware 600.

[0172] The convolution and quantization operations corresponding to the first subtensor a1 and the convolution and quantization operations corresponding to the second subtensor a2 can be performed independently, as shown in Figure 21. Therefore, the software generation unit 325 may assign the divided operations to the neural network hardware 600 by rearranging the order of some of the network (layers).

[0173] The convolution circuit 4 performs the convolution operation of layer 2M-1 corresponding to the first subtensor a1 (shown as layer 2M-1(a1) in Figure 21). Subsequently, the convolution circuit 4 performs the convolution operation of layer 2M-1 corresponding to the second subtensor a2 (shown as layer 2M-1(a2) in Figure 21). In addition, the quantization circuit 5 performs the quantization operation of layer 2M corresponding to the first subtensor a1 (shown as layer 2M(a1) in Figure 21). In this way, the NN execution model 100 can perform the convolution operation of layer 2M-1 corresponding to the second subtensor a2 and the quantization operation of layer 2M corresponding to the first subtensor a1 in parallel.

[0174] Next, the convolution circuit 4 performs the convolution operation on layer 2M+1 corresponding to the first subtensor a1 (shown as layer 2M+1(a1) in Figure 21). The quantization circuit 5 then performs the quantization operation on layer 2M corresponding to the second subtensor a2 (shown as layer 2M(a2) in Figure 21). In this way, the NN execution model 100 can perform the convolution operation on layer 2M+1 corresponding to the first subtensor a1 and the quantization operation on layer 2M corresponding to the second subtensor a2 in parallel.

[0175] By dividing the input data a into subtensors, the neural network hardware 600 can operate the convolutional operation circuit 4 and the quantization operation circuit 5 in parallel. As a result, the waiting time for the convolutional operation circuit 4 and the quantization operation circuit 5 is reduced, improving the computational processing efficiency of the neural network hardware 600. In the example shown in Figure 21, the number of subtensors was 2, but similarly, the neural network hardware 600 can operate the convolutional operation circuit 4 and the quantization operation circuit 5 in parallel even when the number of subtensors is greater than 2.

[0176] As an example of how to perform operations on subtensors, we showed an example (Method 1) in which operations on subtensors in the same layer are performed in the convolution circuit 4 or quantization circuit 5, and then operations on subtensors in the next layer are performed. For example, as shown in Figure 21, in the convolution circuit 4, after performing the convolution operation on layer 2M-1 corresponding to the first subtensor a1 and the second subtensor a2 (operations shown as layer 2M-1(a1) and layer 2M-1(a2) in Figure 21), the convolution operation on layer 2M+1 corresponding to the first subtensor a1 and the second subtensor a2 (operations shown as layer 2M+1(a1) and layer 2M+1(a2) in Figure 21) is performed.

[0177] However, the method of operation on subtensors is not limited to this. The method of operation on subtensors may also be to perform operations on some subtensors in multiple layers and then perform operations on the remaining subtensors (Method 2). For example, in the convolution operation circuit 4, after performing a convolution operation on layer 2M-1 corresponding to the first subtensor a1 and layer 2M+1 corresponding to the first subtensor a1, a convolution operation on layer 2M-1 corresponding to the second subtensor a2 and layer 2M+1 corresponding to the second subtensor a2 may be performed.

[0178] Furthermore, the method for performing operations on a subtensor may also be a combination of Method 1 and Method 2. However, when using Method 2, it is necessary to perform operations according to the dependencies on the order of operations of the subtensor.

[0179] Furthermore, the feasibility of parallel operations on the subtensors described above is determined not only by the dependency on the order of operations of the subtensors, but also by the unused areas in the first memory 1 and the second memory 2. If there are no unused areas in the first memory 1 or the second memory 2 necessary for parallel operations, control is implemented to perform some of the operations in a time-sharing manner instead of in parallel.

[0180] For example, when performing a convolution operation on the same input data a with varying weights w, it is more efficient to perform the convolution operations using the same input data a consecutively. Therefore, the software generation unit 325 rearranges the order of the divided operations so that operations using the same data stored in the first memory 1 and the second memory 2 are performed consecutively as much as possible.

[0181] As described above, the neural network generation method and neural network generation program according to this embodiment can generate a neural network that can be embedded in embedded devices such as IoT devices and can operate with high performance.

[0182] Although the first embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope of the gist of the present invention. Furthermore, the components shown in the above-described embodiment and modified examples can be combined as appropriate.

[0183] (Variation 1) In the above embodiment, the first memory 1 and the second memory 2 were separate memories, but the configurations of the first memory 1 and the second memory 2 are not limited to this. For example, the first memory 1 and the second memory 2 may be the first memory area and the second memory area of ​​the same memory.

[0184] (Modification 2) For example, the data input to the NN execution model 100 and neural network hardware 600 described in the above embodiment is not limited to a single format, but can consist of still images, moving images, audio, text, numerical data, and combinations thereof. Furthermore, the data input to the NN execution model 100 and neural network hardware 600 is not limited to measurement results from physical quantity measuring instruments such as optical sensors, thermometers, Global Positioning System (GPS) measuring instruments, angular velocity measuring instruments, and anemometers that can be mounted on the edge device on which the neural network hardware 600 is installed. It may also be a combination of peripheral information such as base station information, vehicle and ship information, weather information, and congestion information received from peripheral devices via wired or wireless communication, as well as different information such as financial information and personal information.

[0185] (Variation 3) Edge devices equipped with the neural network hardware 600 are envisioned, but not limited to, mobile devices such as mobile phones powered by batteries, smart devices such as personal computers, digital cameras, game consoles, and robotic products. It can also be used in products with high demands for peak power limiting via Power on Ethernet (PoE), reduced product heat generation, or extended operating time, yielding effects not seen in other prior examples. For instance, applying it to in-vehicle cameras mounted on vehicles and ships, or surveillance cameras installed in public facilities and on roads, enables extended recording times while also contributing to weight reduction and increased durability. Similar effects can be achieved by applying it to display devices such as televisions and displays, medical equipment such as medical cameras and surgical robots, and work robots used in manufacturing and construction sites.

[0186] The above-described embodiment may also be realized by recording the program on a computer-readable recording medium, loading the program recorded on this recording medium into a computer system, and executing it. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and storage devices such as hard disks built into a computer system. In addition, "computer-readable recording medium" may include those that dynamically hold programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in such cases. Furthermore, the above-described program may be for realizing only a part of the functions described above, and may also be able to realize the above-described functions in combination with programs already recorded in the computer system.

[0187] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein. [Industrial applicability]

[0188] This invention can be applied to the generation of neural networks. [Explanation of symbols]

[0189] 300 Neural Network Generators 200 Convolutional Neural Networks (CNNs) (NN Functional Models) 100 Neural Network Execution Models (NN Execution Models) 400 Neural Network Hardware Models 500 Software 600 Neural Network Hardware 1 First Memory 2 Second Memory 3. DMA Controller (DMAC) 4. Convolutional arithmetic circuit 42 Multipliers 43. Accumulator Circuit 5 Quantization operation circuit 52 Vector Operation Circuits 53 Quantization circuit 6 Controllers 61 registers PM trained parameters DS training dataset HW Hardware Information NW Network Information OP1 First operator (first function) OP2 Second operator (second function)

Claims

1. A method for generating a neural network execution model in which a computer separately performs convolution and quantization operations from a neural network functional model, The process includes a functional model generation step that generates the neural network functional model using a single first operator that sequentially performs the convolution operation and the quantization operation on the input data. Methods for generating neural networks.

2. The first operator performs a normalization operation in addition to the convolution operation and the quantization operation on the input data. The neural network generation method according to claim 1.

3. The software generation process further comprises generating software for operating the aforementioned neural network execution model, The software causes the neural network execution model to separately execute the convolution operation and the quantization operation included in the first operator in the neural network functional model. The neural network generation method according to claim 1.

4. The functional model generation process generates the neural network functional model using a single second operator that performs the convolution operation on the input data. The second operator is capable of quantizing at least a portion of the convolution operation included in the second operator. A method for generating a neural network according to any one of claims 1 to 3.

5. An execution model generation step, which generates the neural network execution model based on the neural network functional model, A learning process for learning the learning parameters of the neural network execution model, If the learning results of the aforementioned learning process are unacceptable, a neural network functional model regeneration process is performed to regenerate the neural network functional model. Furthermore, The neural network generation method according to claim 1.

6. A program that causes a computer to generate a neural network execution model that separately performs convolution and quantization operations from a neural network functional model, The system includes a functional model generation step in which the computer generates the neural network functional model using a single first operator that sequentially performs the convolution operation and the quantization operation on the input data. A neural network generation program.

7. The first operator performs a normalization operation in addition to the convolution operation and the quantization operation on the input data. The neural network generation program according to claim 6.

8. The software generation process further comprises a step of causing the computer to generate software for operating the neural network execution model, The software causes the neural network execution model to separately execute the convolution operation and the quantization operation included in the first operator in the neural network functional model. The neural network generation program according to claim 6.

9. The functional model generation process involves causing the computer to generate the neural network functional model using a single second operator that performs the convolution operation on the input data. The second operator is capable of quantizing at least a portion of the convolution operation included in the second operator. A neural network generation program according to any one of claims 6 to 8.

10. An execution model generation step in which the computer generates the neural network execution model based on the neural network function model, A learning process in which the computer learns the learning parameters of the neural network execution model, When the learning results of the learning process are unacceptable, a neural network functional model regeneration process is performed to cause the computer to regenerate the neural network functional model; Furthermore, The neural network generation program according to claim 6.

Images