Neural network circuit

Abstract

To provide a highly efficient neural network circuit that can be incorporated in an embedded apparatus such as an IoT apparatus, and a neural network operation method.SOLUTION: A neural network circuit has a convolution operation circuit configured to perform convolution operation relative to input data. The input data is a third or higher order tensor having elements in an x-axis direction, a y-axis direction, and a c-axis direction. The convolution operation circuit includes a first operation unit array that performs convolution operation in the x, y-axis directions and a second operation unit array that performs convolution operation in the c-axis direction relative to an output of the first operation unit array.SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present invention relates to a neural network circuit. [Background technology]

[0002] In recent years, a convolutional neural network (CNN) has been used as a model for image recognition, etc. A neural network circuit that can be embedded in an embedded device such as an IoT device is being used (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-168095 Summary of the Invention [Problem to be solved by the invention]

[0004] There is a demand for neural network circuits that can be embedded in embedded devices such as IoT devices and that can perform neural network operations more efficiently.

[0005] In consideration of the above circumstances, an object of the present invention is to provide a highly efficient neural network circuit that can be incorporated into embedded devices such as IoT devices. [Means for solving the problem]

[0006] In order to solve the above problems, the present invention proposes the following means. A neural network circuit according to a first aspect of the present invention includes a convolution circuit that performs a convolution operation on input data, the input data being a third- or higher-order tensor having elements in the x-axis, y-axis, and c-axis directions, and the convolution circuit includes a first calculator array that performs a convolution operation in the x- and y-axis directions, and a second calculator array that performs a convolution operation in the c-axis direction on the output of the first calculator array. Effect of the Invention

[0007] The neural network circuit of the present invention can be incorporated into embedded devices such as IoT devices and is highly efficient. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates a convolutional neural network. [Diagram 2] FIG. 2 is a diagram for explaining a convolution operation performed by a convolution layer of the convolutional neural network. [Diagram 3] FIG. 13 is a diagram for explaining data division in the convolution operation. [Figure 4] 1 is a diagram showing an overall configuration of a neural network circuit according to a first embodiment; [Diagram 5] FIG. 2 is a diagram showing the overall configuration of a neural network calculation core of the neural network circuit. [Figure 6] 4 is a timing chart showing an example of the operation of the neural network processing core. [Figure 7] 13 is a timing chart showing another example of the operation of the neural network processing core. [Figure 8] FIG. 1 illustrates a neural network computing multi-core. [Figure 9] FIG. 2 is an internal block diagram of the DMAC of the neural network circuit. [Figure 10] FIG. 2 is a state transition diagram of a control circuit of the DMAC. [Figure 11]FIG. 2 is an internal block diagram of a convolution operation circuit of the neural network circuit. [Figure 12] FIG. 2 is an internal block diagram of a multiplier in the convolution operation circuit. [Figure 13] FIG. 2 is an internal block diagram of a first multiply-accumulate unit of the multiplier. [Figure 14] FIG. 2 is an internal block diagram of a second multiply-accumulate unit of the multiplier. [Figure 15] FIG. 2 is an internal block diagram of a multiply-add operation unit of the second multiply-add unit. [Figure 16] FIG. 2 is an internal block diagram of an accumulator circuit of the convolution operation circuit. [Figure 17] FIG. 2 is an internal block diagram of an accumulator unit of the accumulator circuit. [Figure 18] FIG. 2 is an internal block diagram of a quantization calculation circuit of the neural network circuit. [Figure 19] FIG. 2 is an internal block diagram of a vector operation circuit and a quantization circuit of the quantization operation circuit. [Figure 20] FIG. 2 is a block diagram of an arithmetic unit of the vector arithmetic circuit. [Figure 21] FIG. 2 is an internal block diagram of a quantization unit of the quantization circuit. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] First embodiment A first embodiment of the present invention will be described with reference to FIGS. 1 is a diagram showing a convolutional neural network 200 (hereinafter, referred to as "CNN 200"). The calculations performed by the neural network circuit 100 (hereinafter, referred to as "NN circuit 100") according to the first embodiment are at least a part of the trained CNN 200 used during inference.

[0010] [CNN200] The CNN 200 is a network with a multi-layer structure including a convolution layer 210 that performs a convolution operation, a quantization operation layer 220 that performs a quantization operation, and an output layer 230. In at least a part of the CNN 200, the convolution layer 210 and the quantization operation layer 220 are alternately connected. The CNN 200 is a model that is widely used for image recognition and video recognition. The CNN 200 may further include a layer having other functions, such as a fully connected layer.

[0011] FIG. 2 is a diagram illustrating the convolution operation performed by the convolution layer 210. The convolution layer 210 performs a convolution operation on the input data a using a first weight wd and a second weight wp. The convolution layer 210 performs a convolution operation called depthwise separable convolution. The convolution layer 210 executes a convolution operation (Depthwise Convolution) 211 in the spatial direction (xy axis direction) and a convolution operation (Pointwise Convolution) 212 in the channel direction (c axis direction).

[0012] The input data a (including activation data and feature maps) to the convolutional layer 210 is multidimensional data such as image data. In this embodiment, the input data a is a three-dimensional tensor consisting of elements (x, y, c). The convolutional layer 210 of the CNN 200 performs a convolution operation on the low-bit input data a. In this embodiment, the elements of the input data a are 8-bit signed integers.

[0013] If the input data input to the CNN 200 has a different format from the input data a to the convolutional layer 210, such as a 32-bit floating-point type, the CNN 200 may further include a layer before the convolutional layer 210 that performs type conversion and quantization.

[0014] The first weight wd and the second weight wp (also called a filter or a kernel) of the convolution layer 210 are multidimensional data having elements that are learnable parameters. The first weight wd and the second weight wp in the trained CNN 200 are trained data.

[0015] The first weight wd is a three-dimensional tensor consisting of elements (i, j, c). In this embodiment, the elements of the first weight wd are 7-bit signed integers.

[0016] The second weight wp is a two-dimensional tensor consisting of elements (c, d). In this embodiment, the elements of the second weight wp are 1-bit signed integers (0, 1), where the value "0" represents +1 and the value "1" represents -1.

[0017] The convolution layer 210 performs the convolution operations shown in Equation 1 to Equation 3, and outputs output data f. Specifically, intermediate data b is calculated by spatial direction convolution operation 211 shown in Equation 2. Next, output data f is output by channel direction convolution operation 212 shown in Equation 3. In Equation 2, s indicates the stride.

[0018]

number

[0019]

number

[0020]

number

[0021] The spatial convolution operation (Depthwise Convolution) 211 performs a convolution operation in the spatial direction (xy axis direction) using K×K first weights wd prepared for each c-axis direction. In this embodiment, K is 3.

[0022] A channel-direction convolution operation (Pointwise Convolution) 212 performs a convolution operation in the channel direction (c-axis direction) using a second weight wp of 1×1×C prepared for each d-axis direction.

[0023] Depthwise Separable Convolution can significantly reduce the amount of computation while maintaining accuracy compared to conventional convolution operations.

[0024] The quantization operation layer 220 performs quantization and the like on the output of the convolution operation output by the convolution layer 210. The quantization operation layer 220 includes a pooling layer 221, a batch normalization layer 222, an activation function layer 223, and a quantization layer 224.

[0025] The pooling layer 221 compresses the output data f of the convolutional layer 210 by performing calculations such as average pooling (Equation 4) and MAX pooling (Equation 5) on the output data f of the convolutional calculation output by the convolutional layer 210. In Equations 4 and 5, u indicates an input tensor, v indicates an output tensor, and T indicates the size of the pooling region. In Equation 5, max is a function that outputs the maximum value of u for the combination of i and j included in T.

[0026]

number

[0027]

number

[0028] The batch normalization layer 222 normalizes the data distribution of the output data of the quantization operation layer 220 and the pooling layer 221, for example, by the operation shown in Equation 6. In Equation 6, u represents an input tensor, v represents an output tensor, α represents a scale, and β represents a bias. In the trained CNN 200, α and β are trained constant vectors.

[0029]

number

[0030] The activation function layer 223 performs an activation function operation such as ReLU (Equation 5) on the output of the quantization operation layer 220, the pooling layer 221, and the batch normalization layer 222. In Equation 7, u is an input tensor and v is an output tensor. In Equation 7, max is a function that outputs the largest numerical value among arguments.

[0031]

number

[0032] The quantization layer 224 performs quantization on the output of the pooling layer 221 and the activation function layer 223 based on the quantization parameter, for example, as shown in Equation 8. The quantization shown in Equation 8 reduces the input tensor u to 2 bits. In Equation 8, q(c) is a vector of quantization parameters. In the trained CNN 200, q(c) is a trained constant vector. The inequality sign "≦" in Equation 6 may be "<".

[0033]

number

[0034] The output layer 230 is a layer that outputs the result of the CNN 200 using an identity function, a softmax function, etc. The layer preceding the output layer 230 may be the convolution layer 210 or the quantization operation layer 220.

[0035] In the CNN 200, the quantized output data of the quantization layer 224 is input to the convolution layer 210, so the load of the convolution calculation in the convolution layer 210 is smaller than that of other convolution neural networks that do not perform quantization.

[0036] [Split convolution operations] FIG. 3 is a diagram for explaining data division for convolution calculation. The NN circuit 100 divides the input data of the convolution operation (Equation 1) of the convolution layer 210 into partial tensors and performs the operation. The method of division into the partial tensors and the number of divisions are not particularly limited. A partial tensor is formed, for example, by dividing the input data a(x+i, y+j, c) into a(x+i, y+j, co). Note that the NN circuit 100 can also perform the operation without dividing the input data of the convolution operation (Equation 1) of the convolution layer 210.

[0037] In input data division for a convolution operation, the variable c in Equation 1 is divided into blocks of size Bc, as shown in Equation 9. Also, the variable d in Equation 1 is divided into blocks of size Bd, as shown in Equation 10. In Equation 9, co is an offset, and ci is an index from 0 to (Bc-1). In Equation 10, do is an offset, and di is an index from 0 to (Bd-1). Note that size Bc and size Bd may be the same.

[0038]

number

[0039]

number

[0040] The input data a(x+i, y+j, c) in Equation 1 to Equation 3 is divided in the c-axis direction by size Bc, and is represented by the divided input data a(x+i, y+j, co). In the following description, the divided input data a is also referred to as "divided input data a[co]". Each element of the divided input data a[co] is represented by input data a(x+i, y+j, ci).

[0041] The first weight wd(i,j,c) in Equation 1 to Equation 3 is divided by the size Bc in the c-axis direction and expressed as the divided first weight wd(i,j,co). In the following description, the divided first weight wd is also referred to as the "divided first weight wd[co]". Each element of the divided first weight wd[co] is expressed as the first weight wd(i,j,ci).

[0042] The second weight wp(c,d) in Equation 1 to Equation 3 is divided by the size Bc in the c-axis direction and the size Bd in the d-axis direction, and is expressed as the divided second weight wp(co,do). In the following description, the divided second weight wp is also referred to as the "divided second weight wp[co,do]". Each element of the divided second weight wp[co,do] is expressed as the second weight wp(ci,di).

[0043] The output data f(x, y, do) divided by the size Bd is calculated by Equations 11 to 13. The divided output data f(x, y, do) are combined to calculate the final output data f(x, y, d).

[0044]

number

[0045]

number

[0046]

number

[0047] [Expansion of Data for Convolution Operation] The NN circuit 100 expands the input data a, the first weight wd, and the second weight wp in the convolution operation of the convolutional layer 210 to perform the convolution operation.

[0048] The divided input data a[co] is expanded into data having Bc elements in the c-axis direction. The elements of the divided input data a[co] are indexed by ci in the c-axis direction (0 ≤ ci < Bc). The divided input data a[co] has as elements the input data from a(x + i, y + j, co×Bc) to a(x + i, y + j, co×Bc+(Bc - 1)).

[0049] The divided first weight wd[co] is expanded into data having Bc elements in the c-axis direction. The elements of the divided first weight wd[co] are indexed by ci (0 ≤ ci < Bc). The divided first weight wd[co] has as elements the first weight from wd(i, j, co×Bc) to wd(i, j, co×Bc+(Bc - 1)).

[0050] The divided second weight wp[co, do] is expanded into data having a size of Bc in the c-axis direction and Bd elements in the d-axis direction. The elements of the divided second weight wp[co, do] are indexed by ci and di (0 ≤ ci < Bc, 0 ≤ di < Bd). The divided second weight wp[co, do] has as elements the second weight from wp(co×Bc, do×Bd) to wp(co×Bc+(Bc - 1), do×Bd+(Bd - 1)).

[0051] [NN Circuit 100] FIG. 4 is a diagram showing the overall configuration of the NN circuit 100 according to the present embodiment. The NN circuit 100 includes a DMA controller 3 (hereinafter also referred to as "DMAC3"), a controller 6, an IFU 7, and at least one neural network arithmetic core 10 (hereinafter also referred to as "NN arithmetic core 10").

[0052] The NN circuit 100 can implement multiple NN processing cores 10. The NN circuit 100 illustrated in FIG. 4 can implement up to four NN processing cores 10. The multiple NN processing cores 10 constitute a "neural network processing multi-core 10M (hereinafter also referred to as "NN processing multi-core 10M")" that cooperates to execute at least some of the processing of the NN 200. In this embodiment, the multiple NN processing cores 10 are daisy-chained. Note that the number of NN processing cores 10 that can be implemented in the NN circuit 100 may be five or more.

[0053] The DMAC 3 is connected to the external bus EB, and transfers data between an external memory 120 such as a DRAM and the NN processing core 10. The DMAC 3 transfers data read from the external memory 120 to any one of the multiple NN processing cores 10. Note that the DMAC 3 may be capable of transferring the same data read from the external memory 120 to the multiple NN processing cores 10, or may be capable of broadcasting the data.

[0054] The controller 6 is connected to an external bus EB, and operates as a slave to the external host CPU 110. The controller 6 includes a bus bridge 60 and a register 61.

[0055] The bus bridge 60 relays bus access from the external bus EB to the internal bus IB, and also relays write and read requests from the external host CPU 110 to the register 61.

[0056] The register 61 has a parameter register and a status register. The parameter register is a register that controls the operation of the NN circuit 100. The status register includes a pointer to the instruction sequence of each module, the number of instructions, and the like, and indicates the status of the NN circuit 100. The status register may also be configured to include a semaphore S. The external host CPU 110 can access the register 61 via the bus bridge 60 of the controller 6.

[0057] The controller 6 is connected to each block of the NN circuit 100 (DMAC 3, IFU 7, NN calculation core 10) via the internal bus IB. The external host CPU 110 can access each block of the NN circuit 100 via the controller 6. For example, the external host CPU 110 can issue commands to the NN calculation core 10 via the controller 6. Furthermore, each block can update a status register (which may include a semaphore S) held by the controller 6 via the internal bus IB. The status register may be configured to be updated via a dedicated wiring connected to each block.

[0058] Based on an instruction from the external host CPU 110, the IFU (Instruction Fetch Unit) 7 reads out instruction commands for each block (DMAC 3, NN calculation core 10) of the NN circuit 100 from the external memory 120 via the external bus EB. The IFU 7 also transfers the read instruction commands to the corresponding blocks (DMAC 3, NN calculation core 10) of the NN circuit 100. In this embodiment, the instruction commands are stored in a compressed state (hereinafter also referred to as "compressed instruction commands") in the external memory 120. The IFU 7 reads out the compressed instruction commands.

[0059] [NN calculation core 10] FIG. 5 is a diagram showing the overall configuration of the NN processing core 10. As shown in FIG. The NN calculation core 10 includes a first memory 1, a second memory 2, a convolution calculation circuit 4, and a quantization calculation circuit 5. The NN calculation core 10 is characterized in that the convolution calculation circuit 4 and the quantization calculation circuit 5 are formed in a loop shape via the first memory 1 and the second memory 2.

[0060] The first memory 1 is a rewritable memory such as a volatile memory constituted by, for example, SRAM (Static RAM). Data is written to and read from the first memory 1 via the DMAC 3 and the internal bus IB. The external host CPU 110 can input and output data to and from the NN calculation core 10 by writing and reading data to and from the first memory 1.

[0061] The first memory 1 is connected to an input port of the convolution operation circuit 4, and the convolution operation circuit 4 can read data from the first memory 1. The first memory 1 is also loop-connected (C1) to an output port of the quantization operation circuit 5, and the quantization operation circuit 5 can write data to the first memory 1. The first memory 1 is also capable of data transfer via an inter-core connection (C2) between the first memory 1 and another NN operation core 10, and the other NN operation core 10 connected to the inter-core connection (C2) can write data to the first memory 1. In this embodiment, a daisy-chain connection is used as an example of the inter-core connection (C2).

[0062] The second memory 2 is a rewritable memory such as a volatile memory constituted by, for example, SRAM (Static RAM). Data is written to and read from the second memory 2 via the DMAC 3 and the internal bus IB. The external host CPU 110 can input and output data to and from the NN calculation core 10 by writing and reading data to and from the second memory 2.

[0063] The second memory 2 is connected to an input port of the quantization operation circuit 5, and the quantization operation circuit 5 can read data from the second memory 2. In addition, the second memory 2 is connected to an output port of the convolution operation circuit 4, and the convolution operation circuit 4 can write data to the second memory 2.

[0064] The convolution operation circuit 4 is a circuit that performs a convolution operation in the convolution layer 210 of the trained CNN 200. The convolution operation circuit 4 reads the input data a stored in the first memory 1, and performs a convolution operation on the input data a. The convolution operation circuit 4 writes output data f of the convolution operation (hereinafter, also referred to as “convolution operation output data”) to the second memory 2.

[0065] The quantization operation circuit 5 is a circuit that performs at least a part of the quantization operation in the quantization operation layer 220 of the trained CNN 200. The quantization operation circuit 5 reads the output data f of the convolution operation stored in the second memory 2, and performs a quantization operation (an operation including at least quantization among pooling, batch normalization, activation function, and quantization) on the output data f of the convolution operation.

[0066] The quantization calculation circuit 5 writes output data of the quantization calculation (hereinafter also referred to as "quantization calculation output data") to the first memory 1 with which it is loop-connected (C1). In addition, the quantization calculation circuit 5 can transfer data to other NN calculation cores 10 via the inter-core connection (C2), and the quantization calculation circuit 5 can output the quantization calculation output data to other NN calculation cores 10 with which it is inter-core-connected (C2).

[0067] Since the NN processing core 10 has the first memory 1, the second memory 2, etc., it is possible to reduce the number of times that duplicate data is transferred in data transfer from an external memory such as a DRAM by the DMAC 3. This makes it possible to significantly reduce the power consumption or processing load caused by memory access.

[0068] [NN calculation core 10 operation example 1] FIG. 6 is a timing chart showing an example of the operation of the NN processing core 10. In FIG. The DMAC 3 stores the input data a of the layer 1 in the first memory 1. The DMAC 3 may divide the input data a of the layer 1 and transfer it to the first memory 1 in accordance with the order of the convolution operation performed by the convolution operation circuit 4.

[0069] The convolution operation circuit 4 reads out the input data a of layer 1 stored in the first memory 1. The convolution operation circuit 4 performs the convolution operation of layer 1 shown in FIG.

[0070] The quantization calculation circuit 5 reads the output data f of layer 1 stored in the second memory 2. The quantization calculation circuit 5 performs a quantization calculation of layer 2 on the output data f of layer 1. The output data of the quantization calculation of layer 2 is stored in the first memory 1.

[0071] The convolution operation circuit 4 reads the output data of the quantization operation of the layer 2 stored in the first memory 1. The convolution operation circuit 4 performs a convolution operation of the layer 3 using the output data of the quantization operation of the layer 2 as input data a. The output data f of the convolution operation of the layer 3 is stored in the second memory 2.

[0072] The convolution operation circuit 4 reads the output data of the quantization operation of the layer 2M-2 (M is a natural number) stored in the first memory 1. The convolution operation circuit 4 performs the convolution operation of the layer 2M-1 using the output data of the quantization operation of the layer 2M-2 as input data a. The output data f of the convolution operation of the layer 2M-1 is stored in the second memory 2.

[0073] The quantization calculation circuit 5 reads the output data f of the layer 2M-1 stored in the second memory 2. The quantization calculation circuit 5 performs the quantization calculation of the layer 2M on the output data f of the 2M-1 layer. The output data of the quantization calculation of the layer 2M is stored in the first memory 1.

[0074] The convolution operation circuit 4 reads the output data of the quantization operation of the layer 2M stored in the first memory 1. The convolution operation circuit 4 performs a convolution operation of the layer 2M+1 using the output data of the quantization operation of the layer 2M as input data a. The output data f of the convolution operation of the layer 2M+1 is stored in the second memory 2.

[0075] The convolution calculation circuit 4 and the quantization calculation circuit 5 alternately perform calculations to advance the calculations of the CNN 200 shown in Fig. 1. In the NN calculation core 10, the convolution calculation circuit 4 performs the convolution calculations of layers 2M-1 and 2M+1 by time sharing. In the NN calculation core 10, the quantization calculation circuit 5 performs the quantization calculations of layers 2M-2 and 2M by time sharing. Therefore, the circuit scale of the NN calculation core 10 is significantly smaller than when a separate convolution calculation circuit 4 and quantization calculation circuit 5 are implemented for each layer.

[0076] The NN processing core 10 performs the calculations of the CNN 200, which is a multi-layer structure of multiple layers, using a circuit formed in a loop. The NN processing core 10 can efficiently use hardware resources due to the loop circuit configuration. Since the NN processing core 10 forms a circuit in a loop, the parameters in the convolution processing circuit 4 and the quantization processing circuit 5, which change in each layer, are updated appropriately.

[0077] When the calculations of the CNN 200 include calculations that cannot be performed by the NN calculation core 10, the NN calculation core 10 transfers intermediate data to an external calculation device such as an external host CPU 110. After the external calculation device performs calculations on the intermediate data, the calculation results by the external calculation device are input to the first memory 1 and / or the second memory 2. The NN calculation core 10 resumes calculations on the calculation results by the external calculation device.

[0078] [NN calculation core 10 operation example 2] FIG. 7 is a timing chart showing another example of the operation of the NN processing core 10. In FIG. The NN processing core 10 may divide the input data a into partial tensors and perform operations on the partial tensors by time division. The method of division into the partial tensors and the number of divisions are not particularly limited.

[0079] 7 shows an example of operation when input data a is decomposed into two partial tensors. The decomposed partial tensors are called "first partial tensor a1" and "second partial tensor a2". For example, the convolution operation of layer 2M-1 is decomposed into a convolution operation corresponding to the first partial tensor a1 (indicated as "layer 2M-1(a1)" in FIG. 7) and a convolution operation corresponding to the second partial tensor a2 (indicated as "layer 2M-1(a2)" in FIG. 7).

[0080] The convolution operation and quantization operation corresponding to the first partial tensor a1 and the convolution operation and quantization operation corresponding to the second partial tensor a2 can be performed independently, as shown in FIG.

[0081] The convolution operation circuit 4 performs a convolution operation of the layer 2M-1 corresponding to the first partial tensor a1 (operation shown as layer 2M-1(a1) in FIG. 7). After that, the convolution operation circuit 4 performs a convolution operation of the layer 2M-1 corresponding to the second partial tensor a2 (operation shown as layer 2M-1(a2) in FIG. 7). In addition, the quantization operation circuit 5 performs a quantization operation of the layer 2M corresponding to the first partial tensor a1 (operation shown as layer 2M(a1) in FIG. 7). In this way, the NN operation core 10 can perform the convolution operation of the layer 2M-1 corresponding to the second partial tensor a2 and the quantization operation of the layer 2M corresponding to the first partial tensor a1 in parallel.

[0082] Next, the convolution operation circuit 4 performs a convolution operation of layer 2M+1 corresponding to the first partial tensor a1 (operation shown as layer 2M+1(a1) in FIG. 7). Moreover, the quantization operation circuit 5 performs a quantization operation of layer 2M corresponding to the second partial tensor a2 (operation shown as layer 2M(a2) in FIG. 7). In this way, the NN operation core 10 can perform, in parallel, the convolution operation of layer 2M+1 corresponding to the first partial tensor a1 and the quantization operation of layer 2M corresponding to the second partial tensor a2.

[0083] The convolution operation and quantization operation corresponding to the first partial tensor a1 and the convolution operation and quantization operation corresponding to the second partial tensor a2 can be performed independently. Therefore, the NN processing core 10 may perform, for example, the convolution operation of the layer 2M-1 corresponding to the first partial tensor a1 and the quantization operation of the layer 2M+2 corresponding to the second partial tensor a2 in parallel. In other words, the convolution operation and quantization operation that the NN processing core 10 performs in parallel are not limited to the operations of consecutive layers.

[0084] By dividing the input data a into partial tensors, the NN processing core 10 can operate the convolution processing circuit 4 and the quantization processing circuit 5 in parallel. As a result, the waiting time of the convolution processing circuit 4 and the quantization processing circuit 5 is reduced, improving the processing efficiency of the NN processing core 10. In the operation example shown in FIG. 7, the number of divisions is 2, but even when the number of divisions is greater than 2, the NN processing core 10 can similarly operate the convolution processing circuit 4 and the quantization processing circuit 5 in parallel.

[0085] For example, when the input data a is divided into a “first partial tensor a1”, a “second partial tensor a2”, and a “third partial tensor a3”, the NN processing core 10 may perform a convolution operation of the layer 2M-1 corresponding to the second partial tensor a2 and a quantization operation of the layer 2M corresponding to the third partial tensor a3 in parallel. The order of the operations is appropriately changed depending on the storage status of the input data a in the first memory 1 and the second memory 2.

[0086] As an example of a method of computing partial tensors, an example (method 1) has been shown in which a partial tensor in the same layer is computed by the convolution computation circuit 4 or the quantization computation circuit 5, and then a partial tensor in the next layer is computed. For example, as shown in Fig. 7, in the convolution computation circuit 4, a convolution computation of layer 2M-1 corresponding to the first partial tensor a1 and the second partial tensor a2 (computation shown by layer 2M-1(a1) and layer 2M-1(a2) in Fig. 7) is performed, and then a convolution computation of layer 2M+1 corresponding to the first partial tensor a1 and the second partial tensor a2 (computation shown by layer 2M+1(a1) and layer 2M+1(a2) in Fig. 7).

[0087] However, the method of operation on the partial tensors is not limited to this. The method of operation on the partial tensors may be a method of performing an operation on some of the partial tensors in multiple layers and then performing an operation on the remaining partial tensors (method 2). For example, in the convolution operation circuit 4, after performing a convolution operation on the layer 2M-1 corresponding to the first partial tensor a1 and the layer 2M+1 corresponding to the first partial tensor a1, a convolution operation on the layer 2M-1 corresponding to the second partial tensor a2 and the layer 2M+1 corresponding to the second partial tensor a2 may be performed.

[0088] Furthermore, the method of computing partial tensors may be a method of computing partial tensors by combining method 1 and method 2. However, when using method 2, it is necessary to perform the computation in accordance with the dependency relationship regarding the computation order of the partial tensors.

[0089] [NN calculation multi-core 10M] FIG. 8 is a diagram showing the NN calculation multi-core 10M. The NN processing multi-core 10M illustrated in Fig. 8 includes two daisy-chained NN processing cores 10. When distinguishing between the two NN processing cores 10, the two NN processing cores 10 are referred to as a "first NN processing core 10A" and a "second NN processing core 10B." In Fig. 8, the first memory 1 is abbreviated as "A," the convolution processing circuit 4 as "C," the second memory 2 as "F," and the quantization processing circuit 5 as "Q."

[0090] Specifically, the quantization calculation circuit 5 of the first NN arithmetic core 10A and the first memory 1 of the second NN arithmetic core 10B are daisy-chain connected (C2). The quantization calculation circuit 5 of the first NN arithmetic core 10A can write quantization calculation output data to the first memory 1 of the first NN arithmetic core 10A connected in a loop (C1) and / or the first memory 1 of the second NN arithmetic core 10B connected in a daisy chain (C2).

[0091] Specifically, the quantization calculation circuit 5 of the second NN calculation core 10B and the first memory 1 of the first NN calculation core 10A are daisy-chain connected (C2). The quantization calculation circuit 5 of the second NN calculation core 10B can write the quantization calculation output data to the first memory 1 of the second NN calculation core 10B connected in a loop (C1) and / or the first memory 1 of the first NN calculation core 10A connected in a daisy chain (C2).

[0092] Similarly, when the NN processing multi-core 10M has three or more NN processing cores 10, the multiple NN processing cores 10 are connected in a daisy chain. The quantization processing circuit 5 of the NN processing cores 10 other than the final stage NN processing core 10 is connected in a daisy chain (C2) with the first memory 1 of the subsequent stage NN processing core 10B. The quantization processing circuit 5 of the final stage NN processing core 10 is connected in a daisy chain (C2) with the first memory 1 of the first stage NN processing core 10. The multiple NN processing cores 10 are characterized by being formed in a daisy chain loop (linked together).

[0093] In one NN calculation core 10, the first memory (A) 1, the convolution calculation circuit (C) 4, the second memory (F) 2, and the quantization calculation circuit (Q) 5 are connected in a loop. On the other hand, in the NN calculation multi-core 10M, the first memory (A) 1, the convolution calculation circuit (C) 4, the second memory (F) 2, and the quantization calculation circuit (Q) 5 are connected in a daisy chain loop (linked together) so that the first memory (A) 1, the convolution calculation circuit (C) 4, the second memory (F) 2, and the quantization calculation circuit (Q) 5 are repeatedly arranged in the same order.

[0094] The multiple NN calculation cores 10 constituting the NN calculation multi-core 10M do not need to have the same hardware configuration. For example, the capacity and configuration of the first memory 1 of the first NN calculation core 10A may be different from the capacity and configuration of the first memory 1 of the second NN calculation core 10B. For example, the configuration of the quantization calculation circuit 5 of the first NN calculation core 10A may be different from the configuration of the quantization calculation circuit 5 of the second NN calculation core 10B.

[0095] Next, each component of the NN circuit 100 will be described in detail.

[0096] [DMAC3] FIG. 9 is an internal block diagram of the DMAC3. The DMAC 3 has a data transfer circuit 31 and a state controller 32. The DMAC 3 has a state controller 32 dedicated to the data transfer circuit 31, and when an instruction command is input, the DMAC 3 can perform DMA data transfer without requiring an external controller.

[0097] The data transfer circuit 31 is connected to the external bus EB, and performs DMA data transfer between an external memory 120 such as a DRAM and the NN processing core 10. The number of DMA channels of the data transfer circuit 31 is not limited. For example, the first NN processing core 10A and the second NN processing core 10B may each have a dedicated DMA channel.

[0098] The state controller 32 controls the state of the data transfer circuit 31. The state controller 32 is also connected to the controller 6 via an internal bus IB. The state controller 32 has an instruction queue 33 and a control circuit .

[0099] The instruction queue 33 is a queue that stores instruction commands C3 for the DMAC 3, and is configured, for example, by a FIFO memory. One or more instruction commands C3 are written to the instruction queue 33 via the IFU 7 or the internal bus IB.

[0100] The control circuit 34 is a state machine that decodes the instruction command C3 and sequentially controls the data transfer circuit 31 based on the instruction command C3. The control circuit 34 may be implemented by a logic circuit or a CPU controlled by software.

[0101] FIG. 10 is a state transition diagram of the control circuit 34. When an instruction command C3 is input to the instruction queue 33 (Not empty), the control circuit 34 transitions from the idle state ST1 to the decode state ST2.

[0102] In the decode state ST2, the control circuit 34 decodes the instruction command C3 output from the instruction queue 33. The control circuit 34 also reads the semaphore S stored in the register 61 of the controller 6, and determines whether the operation of the data transfer circuit 31 instructed in the instruction command C3 is executable. If it is not executable (Not ready), the control circuit 34 waits until it is executable (Wait). If it is executable (Ready), the control circuit 34 transitions from the decode state ST2 to the execution state ST3.

[0103] In the execution state ST3, the control circuit 34 controls the data transfer circuit 31 to cause the data transfer circuit 31 to perform the operation instructed in the instruction command C3. When the operation of the data transfer circuit 31 is completed, the control circuit 34 removes the executed instruction command C3 from the instruction queue 33 and updates the semaphore S stored in the register 61 of the controller 6. If there are instructions in the instruction queue 33 (Not empty), the control circuit 34 transitions from the execution state ST3 to the decode state ST2. If there are no instructions in the instruction queue 33 (Empty), the control circuit 34 transitions from the execution state ST3 to the idle state ST1.

[0104] [Convolution circuit 4] FIG. 11 is an internal block diagram of the convolution operation circuit 4. As shown in FIG. The convolution operation circuit 4 has a weight memory 41, a multiplier 42, an accumulator circuit 43, and a state controller 44. The convolution operation circuit 4 has a state controller 44 dedicated to the multiplier 42 and the accumulator circuit 43, and when an instruction command is input, the convolution operation can be performed without requiring an external controller.

[0105] The weight memory 41 is a memory in which the first weight wd and the second weight wp used in the convolution calculation are stored, and is a rewritable memory such as a volatile memory constituted by, for example, an SRAM (Static RAM), etc. The DMAC 3 writes the first weight wd and the second weight wp required for the convolution calculation into the weight memory 41 by DMA transfer.

[0106] FIG. 12 is an internal block diagram of the multiplier 42. The multiplier 42 has a first multiplier (first calculator array) 421 and a second multiplier (second calculator array) 422. The first multiplier 421 performs a spatial direction convolution operation (Depthwise Convolution) 211 shown in Equation 12. The second multiplier 422 performs a channel direction convolution operation (Pointwise Convolution) 212 shown in Equation 13.

[0107] The first multiplier (first calculator array) 421 multiplies each element a(x+i,y+j,ci) of the divided input data a(x+i,y+j,co) by each element wd(i,j,ci) of the divided first weight wd(i,j,co). The first multiplier 421 has Bc first product-accumulator units 421A, and can multiply the elements a(x+i,y+j,ci) of the divided input data a(x+i,y+j,co) by the elements wd(i,j,ci) of the divided first weight wd(i,j,co) in parallel.

[0108] FIG. 13 is an internal block diagram of the first product-sum unit 421A. The first product-sum unit 421A multiplies an element a(x+i, y+j, ci) by an element wd(i, j, ci). The first product-sum unit 421A has a product-sum operation unit 47 capable of multiplying a K×K element a by a K×K element wd. In this embodiment, the 3×3 elements of the element a and the weight wd on the xy-axis plane are multiplied by nine product-sum operation units 47 to output 1×1 operation data Od(ci). The operation data Od(ci) is, for example, 16 bits or 8 bits.

[0109] The first multiply-accumulate unit 421A has an input register 49 in which the element a is stored. The input register 49 has a first input register 49a, a second input register 49b, and a third input register 49c.

[0110] For example, the y-axis coordinate of the first input register 49a is (y), and 32 elements a(x+i, y, ci) that are consecutive in the x-axis direction are input (0≦i<32). This data is called "line data L(y)".

[0111] The second input register 49b has, for example, a y-axis coordinate of (y+1), and receives 32 consecutive elements a(x+i, y+1, ci) in the x-axis direction (0≦i<32). This data is called "line data L(y+1)".

[0112] For example, the third input register 49c has a y-axis coordinate of (y+2), and receives 32 consecutive elements a(x+i, y+2, ci) in the x-axis direction (0≦i<32). This data is called "line data L(y+2)".

[0113] The line data L(y), line data L(y+1), and line data L(y+2) are continuous line data in the y-axis direction on the xy-axis plane. The multiple line data L input to the input register 49 do not have to be continuous line data in the y-axis direction on the xy-axis plane. The multiple line data L input to the input register 49 may be line data spaced a predetermined number of lines apart in the y-axis direction, or may be continuous line data in the x-axis direction.

[0114] By shifting the data in the input register 49, the element a to be multiplied with the element wd of K×K is switched.

[0115] The first multiplier 421 concatenates the operation data Od(ci) output from the c first product-accumulator units 421A in the c-axis direction, and outputs intermediate data b(x, y, ci) of 1×1×Bc.

[0116] 12, the second multiplier (second calculator array) 422 has Bd second product-sum units 422A. The same intermediate data b(x, y, ci) is input to the Bd second product-sum units 422A. Each element wp(ci, di) of the divided second weight wp(co, do) is input to the Bd second product-sum units 422A.

[0117] FIG. 14 is an internal block diagram of the second multiply-accumulate unit 422A. The second product-sum unit 422A adds up the multiplication results obtained by multiplying each element of the intermediate data b(x, y, ci) by the second weight wp(ci, di). The second product-sum unit 422A has Bc product-sum operation units 47. The Bc product-sum operation units 47 are connected in series. Each element b(ci) of the intermediate data b(x, y, ci) is input to the Bc product-sum operation units 47.

[0118] In this embodiment, the number of the first multipliers 421 and the second multipliers 422 is matched to the number of divisions of the divided input data a based on the size Bd and the size Bc, and the first multipliers 421 and the second multipliers 422 can perform parallel operation on the divided input data a with high efficiency. The first multipliers 421 and the second multipliers 422 may further parallelize the parallel operation in the xy direction. Specifically, the first multipliers 421 and the second multipliers 422 may include a plurality of units, with the first multiplier 421 having Bc first product-accumulator units 421A and the second multiplier 422 having Bd second product-accumulator units 422A as one unit. The first multipliers 421 and the second multipliers 422 can realize parallel operation in the xy direction for adjacent rows or adjacent columns held in the first memory 1. In order to perform parallelization, it is desirable that the number of the plurality of units is a power of 2, such as 4 or 8, taking into consideration reading from the first memory 1.

[0119] FIG. 15 is an internal block diagram of the product-sum calculation unit 47. As shown in FIG. The product-sum operation unit 47 multiplies an element b(ci) of the intermediate data b(x, y, ci) by a second weight wp(ci, di). The product-sum operation unit 47 also adds the multiplication result to the multiplication result S(ci, di) of another product-sum operation unit 47. The product-sum operation unit 47 outputs the addition result S(ci+1, di).

[0120] The bit precision of the second weight wp is preferably lower than that of the first weight wd. The first product-sum unit 421A adds nine product-sum operation results together to output the output Od(ci), but the second product-sum unit 422A needs to add Bc product-sum operation results together. For example, if Bc is 64, the second product-sum unit 422A adds 64 product-sum operations together to output the multiplication result S(ci,di). In other words, the second product-sum unit 422A adds a larger number of operations at once, so there is a higher risk of the addition result overflowing. For this reason, the bit precision of the second weight wp is preferably set to a low bit.

[0121] The multiply-accumulate unit 47 has an inverter 47a, a selector 47b, and an adder 47c. The multiply-accumulate unit 47 performs multiplication using only the inverter 47a and the selector 47b without using a multiplier. When the second weight wp(di) is "0", the selector 47b selects the input of the element b(ci). When the second weight wp(di) is "1", the selector 47b selects the complement of the element b(ci) inverted by the inverter. The second weight wp(di) is also input to the carry-in of the adder 47c. When the second weight wp(di) is "0", the adder 47c outputs a value obtained by adding the element b(ci) to S(ci,di). When the second weight wp(,di) is "1", the adder 47c outputs a value obtained by subtracting the element b(ci) from S(ci,di).

[0122] In this embodiment, the first product-accumulator unit 421A has an input register 49 capable of holding three lines of line data, and processes the three lines of data in parallel. The first product-accumulator unit 421A may share the input register 49 of the first product-accumulator unit 421A with another first product-accumulator unit 421A. As an example, the first product-accumulator unit 421A may further include a fourth input register 49d to which "line data L(y+3)" is input, which is 32 elements a(x+i, y+1, ci) whose y-axis coordinate is (y+3) and which are continuous in the x-axis direction. In this case, the second input register 49b, the third input register 49c, and the fourth input register 49d are shared with different first product-accumulator units 421A. In this case, the two first product-accumulator units 421A can process four rows of line data in parallel. The number of required input registers is reduced from six to four. Similarly, if the four first multiply-accumulate units 421A share an input register, the number of required input registers is reduced from twelve to six.

[0123] FIG. 16 is an internal block diagram of the accumulator circuit 43. The accumulator circuit 43 accumulates the product-sum operation results O(di) of the multiplier 42 in the second memory 2. The accumulator circuit 43 has Bd accumulator units 48 and can accumulate the Bd product-sum operation results O(di) in parallel in the second memory 2.

[0124] FIG. 17 is an internal block diagram of the accumulator unit 48. The accumulator unit 48 has an adder 48a and a mask section 48b. The adder 48a adds an element O(di) of the product-sum operation result O to a partial sum that is an intermediate step of the convolution operation shown in Equation 1 to Equation 3 stored in the second memory 2. The addition result is 16 bits per element. The addition result is not limited to 16 bits per element, and may be, for example, 15 bits or 17 bits per element.

[0125] The adder 48a writes the addition result to the same address in the second memory 2. When the initialization signal clear is asserted, the masking unit 48b masks the output from the second memory 2 and sets the addition target for the element O(di) to zero. The initialization signal clear is asserted when no intermediate partial sums are stored in the second memory 2.

[0126] When the convolution operation by the multiplier 42 and the accumulator circuit 43 is completed, the output data f(x, y, do) is stored in the second memory 2.

[0127] The state controller 44 controls the states of the multiplier 42 and the accumulator circuit 43. The state controller 44 is also connected to the controller 6 via an internal bus IB. The state controller 44 has an instruction queue 45 and a control circuit 46.

[0128] The instruction queue 45 is a queue that stores the instruction commands C4 for the convolution operation circuit 4, and is configured by, for example, a FIFO memory. The instruction commands C4 are written to the instruction queue 45 via the IFU 7 or the internal bus IB.

[0129] The control circuit 46 is a state machine that decodes the instruction command C4 and controls the multiplier 42 and the accumulator circuit 43 based on the instruction command C4. The control circuit 46 has a similar configuration to the control circuit 34 of the state controller 32 of the DMAC 3.

[0130] [Quantization operation circuit 5] FIG. 18 is an internal block diagram of the quantization calculation circuit 5. The quantization calculation circuit 5 has a quantization parameter memory 51, a vector calculation circuit 52, a quantization circuit 53, and a state controller 54. The quantization calculation circuit 5 has a state controller 54 dedicated to the vector calculation circuit 52 and the quantization circuit 53, and when an instruction command is input, the quantization calculation can be performed without requiring an external controller.

[0131] The quantization parameter memory 51 is a memory in which the quantization parameter q used in the quantization operation is stored, and is a rewritable memory such as a volatile memory constituted by, for example, an SRAM (Static RAM), etc. The DMAC 3 writes the quantization parameter q required for the quantization operation into the quantization parameter memory 51 by DMA transfer.

[0132] FIG. 19 is an internal block diagram of the vector calculation circuit 52 and the quantization circuit 53. The vector operation circuit 52 performs an operation on the output data f(x, y, do) stored in the second memory 2. The vector operation circuit 52 has Bd operation units 57, and performs SIMD operations in parallel on the output data f(x, y, do).

[0133] FIG. 20 is a block diagram of the arithmetic unit 57. The arithmetic unit 57 includes, for example, an ALU 57a, a first selector 57b, a second selector 57c, a register 57d, and a shifter 57e. The arithmetic unit 57 may further include other arithmetic units included in a known general-purpose SIMD arithmetic circuit.

[0134] The vector calculation circuit 52 performs at least one of the calculations of the pooling layer 221, the batch normalization layer 222, and the activation function layer 223 in the quantization calculation layer 220 on the output data f(x, y, do) by combining the calculation units and the like contained in the calculation unit 57.

[0135] The arithmetic unit 57 can add the data stored in the register 57d and the element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a. The arithmetic unit 57 can store the addition result by the ALU 57a in the register 57d. The arithmetic unit 57 can initialize the addition result by inputting "0" to the ALU 57a instead of the data stored in the register 57d by the selection of the first selector 57b. For example, when the pooling area is 2×2, the shifter 57e can output the average value of the addition result by shifting the output of the ALU 57a to the right by 2 bits. The vector arithmetic circuit 52 can perform the average pooling calculation shown in Equation 2 by repeating the above calculations by the Bd arithmetic units 57.

[0136] The arithmetic unit 57 can compare the data stored in the register 57d with the element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a. The arithmetic unit 57 controls the second selector 57c according to the comparison result by the ALU 57a, and can select the larger of the data stored in the register 57d and the element f(di). The arithmetic unit 57 can initialize the comparison target to the minimum value by inputting the minimum value of the possible values ​​of the element f(di) to the ALU 57a by the selection of the first selector 57b. In this embodiment, the element f(di) is a 16-bit signed integer, so the minimum value of the possible values ​​of the element f(di) is "0x8000". The vector arithmetic circuit 52 can perform the MAX pooling calculation of Equation 3 by repeating the above calculations by the Bd arithmetic units 57. Note that in the MAX pooling calculation, the shifter 57e does not shift the output of the second selector 57c.

[0137] The arithmetic unit 57 can subtract data stored in the register 57d and an element f(di) of the output data f(x, y, do) read from the second memory 2 by the ALU 57a. The shifter 57e can shift the output of the ALU 57a to the left (i.e., multiplication) or to the right (i.e., division). The vector arithmetic circuit 52 can perform the batch normalization calculation of Equation 4 by repeating the above calculations by the Bd arithmetic units 57.

[0138] The arithmetic unit 57 can compare the element f(di) of the output data f(x, y, do) read from the second memory 2 with "0" selected by the first selector 57b by the ALU 57a. The arithmetic unit 57 can select and output either the element f(di) or a constant value "0" previously stored in the register 57d according to the comparison result by the ALU 57a. The vector arithmetic circuit 52 can perform the ReLU arithmetic of Equation 5 by repeating the above arithmetic operations by the Bd arithmetic units 57.

[0139] The vector operation circuit 52 can perform average pooling, MAX pooling, batch normalization, activation function operations, and combinations of these operations. Since the vector operation circuit 52 can perform general-purpose SIMD operations, it may perform other operations necessary for the operations in the quantization operation layer 220. In addition, the vector operation circuit 52 may perform operations other than those in the quantization operation layer 220.

[0140] It is to be noted that the quantization calculation circuit 5 does not have to include the vector calculation circuit 52. When the quantization calculation circuit 5 does not include the vector calculation circuit 52, the output data f(x, y, do) is input to the quantization circuit 53.

[0141] The quantization circuit 53 quantizes the output data of the vector operation circuit 52. The quantization circuit 53 has Bd quantization units 58, as shown in FIG.

[0142] FIG. 21 is an internal block diagram of the quantization unit 58. The quantization unit 58 quantizes the elements in(di) of the output data of the vector operation circuit 52. The quantization unit 58 includes a comparator 58a and an encoder 58b. The quantization unit 58 performs the operation (Equation 6) of the quantization layer 224 in the quantization operation layer 220 on the output data (16 bits / element) of the vector operation circuit 52. The quantization unit 58 reads the necessary quantization parameters q(th0, th1, th2) from the quantization parameter memory 51, and the comparator 58a compares the input in(di) with the quantization parameters q. The quantization unit 58 quantizes the comparison result by the comparator 58a to 2 bits / element by the encoder 58b. The quantization unit 58 may quantize the elements in(di) of the output data of the vector operation circuit 52 to a number of bits other than 2 bits (for example, 8 bits). Since α(c) and β(c) in Equation 4 are different parameters for each variable c, the quantization parameters q(th0, th1, th2) reflecting α(c) and β(c) are different parameters for each in(di).

[0143] The quantization unit 58 classifies the input in(di) into four regions (for example, in≦th0, th0<in≦th1, th1<in≦th2, th2<in) by comparing the input in(di) with three threshold values th0, th1, th2, encodes the classification result into 2 bits, and outputs it. The quantization unit 58 can also perform operations of Batch Normalization and activation functions together with quantization by setting the quantization parameters q(th0, th1, th2).

[0144] The quantization unit 58 can perform the operation of Batch Normalization shown in Equation 4 together with quantization by setting the threshold value th0 as β(c) in Equation 4 and the threshold differences (th1−th0) and (th2−th1) as α(c) in Equation 4. By increasing (th1−th0) and (th2−th1), α(c) can be decreased. By decreasing (th1−th0) and (th2−th1), α(c) can be increased.

[0145] The quantization unit 58 can perform an activation function in conjunction with quantization of the input in(di). For example, the quantization unit 58 saturates the output value in regions where in(di) ≤ th0 and th2 < in(di). The quantization unit 58 can perform the operation of the activation function in conjunction with quantization by setting the quantization parameter q so that the output is non-linear.

[0146] The state controller 54 controls the states of the vector operation circuit 52 and the quantization circuit 53. The state controller 54 is also connected to the controller 6 via the internal bus IB. The state controller 54 includes an instruction queue 55 and a control circuit 56.

[0147] The instruction queue 55 is a queue in which the instruction command C5 for the quantization operation circuit 5 is stored, and is configured by, for example, a FIFO memory. The instruction command C5 is written to the instruction queue 55 via the IFU7 or the internal bus IB.

[0148] The control circuit 56 is a state machine that decodes the instruction command C5 and controls the vector operation circuit 52 and the quantization circuit 53 based on the instruction command C5. The control circuit 56 has the same configuration as the control circuit 34 of the state controller 32 of the DMAC3.

[0149] [Controller 6] The controller 6 transfers the instruction commands transferred from the external host CPU 110 to the instruction queues of the DMAC3, the convolution operation circuit 4, and the quantization operation circuit 5 via the internal bus IB. The controller 6 may have an instruction memory that stores instruction commands for each circuit.

[0150] The controller 6 is connected to an external bus EB, and operates as a slave to the external host CPU 110. The controller 6 has registers 61 including a parameter register and a status register. The parameter register is a register that controls the operation of the NN circuit 100. The status register is a register that indicates the status of the NN circuit 100, including the semaphore S.

[0151] According to the neural network circuit 100 of this embodiment, the NN circuit 100 that can be embedded in an embedded device such as an IoT device can operate with high performance. Since the amount of calculation of the convolution calculation circuit 4 is significantly less than that of a general convolution calculation circuit, the neural network circuit 100 can execute neural network calculations with higher efficiency.

[0152] Although the first embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and design modifications and the like that do not depart from the gist of the present invention are also included. Also, the components shown in the above embodiment and modified examples can be appropriately combined to form a configuration.

[0153] (Variation 1) In the above embodiment, the first memory 1 and the second memory 2 are separate memories, but the aspects of the first memory 1 and the second memory 2 are not limited to this. The first memory 1 and the second memory 2 may be, for example, a first memory area and a second memory area in the same memory.

[0154] (Variation 2) For example, the data input to the NN circuit 100 described in the above embodiment is not limited to a single format, and can be composed of still images, moving images, sounds, characters, numbers, or combinations of these. The data input to the NN circuit 100 is not limited to the measurement results of physical quantity measuring instruments such as optical sensors, thermometers, Global Positioning System (GPS) measuring instruments, angular velocity measuring instruments, and anemometers that may be mounted on the edge device in which the NN circuit 100 is provided. Different information such as base station information received from peripheral devices via wired or wireless communication, information on vehicles and ships, weather information, and congestion information, as well as peripheral information, financial information, and personal information may also be combined.

[0155] (Variation 3) The edge device in which the NN circuit 100 is provided is assumed to be a communication device such as a battery-powered mobile phone, a smart device such as a personal computer, a digital camera, a game device, a robot product, and other mobile devices, but is not limited thereto. By using the NN circuit 100 in products that have a high demand for limiting the peak power that can be supplied by Power on Ethernet (PoE), reducing heat generation of the product, or long-term operation, it is possible to obtain effects not seen in other prior art. For example, by applying the NN circuit 100 to an in-vehicle camera mounted on a vehicle or ship, or a surveillance camera installed in a public facility or on the road, it is possible to realize long-term shooting, and it also contributes to weight reduction and high durability. In addition, the same effect can be obtained by applying the NN circuit 100 to a display device such as a television or a display, medical equipment such as a medical camera or a surgical robot, and a work robot used in a manufacturing site or a construction site.

[0156] (Variation 4) The NN circuit 100 may be realized in part or in whole by using one or more processors. For example, the NN circuit 100 may realize in part or in whole the input layer or the output layer by software processing by a processor. A part of the input layer or the output layer realized by software processing is, for example, data normalization or conversion. This makes it possible to support various types of input formats or output formats. Note that the software executed by the processor may be configured to be rewritable using communication means or external media.

[0157] (Variation 5) The NN circuit 100 may implement part of the processing in the CNN 200 by combining a graphics processing unit (GPU) or the like on the cloud. The NN circuit 100 can implement more complex processing with fewer resources by performing further processing on the cloud in addition to the processing performed on the edge device on which the NN circuit 100 is installed, or by performing processing on the edge device in addition to the processing on the cloud. With such a configuration, the NN circuit 100 can reduce the amount of communication between the edge device and the cloud by distributing processing.

[0158] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects. [Industrial Applicability]

[0159] The present invention can be applied to neural network operations. [Explanation of symbols]

[0160] 200 Convolutional Neural Networks 100 Neural network circuit (NN circuit) 10 Neural network calculation core (NN calculation core) 10A First Neural Network Calculation Core (First NN Calculation Core) 10B Second Neural Network Calculation Core (Second NN Calculation Core) 10M Neural network calculation multi-core (NN calculation multi-core) 1. First Memory 2 Second Memory 3. DMA Controller (DMAC) 4. Convolution Circuit 42 Multiplier 43 Accumulator Circuit 5 Quantization operation circuit 52 Vector arithmetic circuit 53 Quantization circuit 6 Controller 61 Registers 7 IFU

Claims

1. A convolution operation circuit is provided for performing a convolution operation on input data, The input data is a third- or higher-order tensor having elements in the x-axis, y-axis, and c-axis directions; The convolution operation circuit includes: a first calculator array for performing convolution calculations in the x- and y-axis directions; a second calculator array that performs a convolution operation in a c-axis direction on an output of the first calculator array; having Neural network circuit.

2. the input data input to the convolution operation circuit is divided by a first size in the c-axis direction; the first arithmetic unit array has first multiply-accumulate units at least as many as the first size; 2. The neural network circuit of claim 1.

3. the input data is a fourth- or higher-order tensor having elements in the x-axis direction, the y-axis direction, the c-axis direction, and the d-axis direction; the input data input to the convolution operation circuit is divided by a second size in the d-axis direction; the second calculator array has second multiply-accumulate units at least as many as the second size; 3. The neural network circuit according to claim 1 or 2.

4. the first multiply-accumulate unit has an input register to which the input data is input, and a multiply-accumulate operation unit that operates a plurality of elements of the input data in the x-axis direction and the y-axis direction; 3. The neural network circuit of claim 2.

5. the input register stores line data that are continuous in the x-axis direction, and stores the line data that are spaced apart by a predetermined number of lines in the y-axis direction; 5. The neural network circuit of claim 4.

6. a first weight used by the first calculator array for the convolution operation and a second weight used by the second calculator array for the convolution operation are input to the convolution operation circuit; 2. The neural network circuit of claim 1.

7. the bit precision of the second weight is lower than the bit precision of the first weight; 7. The neural network circuit of claim 6.

8. A method for performing a convolution operation on input data that is a third- or higher-order tensor having elements in the x-axis, y-axis, and c-axis directions, comprising the steps of: A first calculation step of performing a convolution calculation in the x-axis and y-axis directions; a second calculation step of performing a convolution operation in the c-axis direction on the output of the first calculation step; having Neural network calculation method.

Images