Neural network circuit and method for controlling a neural network circuit

Abstract

To enable high-performance operation of neural network circuits that can be embedded in embedded devices such as IoT devices. [Solution] The neural network circuit comprises a convolution circuit that performs a convolution operation on input data, and a quantization circuit that performs a quantization operation on the output data of the convolution operation of the convolution circuit, wherein the convolution circuit enables clock gating of a first clock supplied to at least a portion of the convolution circuit when waiting for the execution of the convolution operation.

Description

Technical Field

[0001] The present invention relates to a neural network circuit and a method for controlling a neural network circuit.

Background Art

[0002] In recent years, convolutional neural networks (CNNs) have been used as models for image recognition and the like. A convolutional neural network has a multi-layer structure including convolutional layers and pooling layers, and requires a large number of operations such as convolutional operations. Various operation methods for accelerating the operations by a convolutional neural network have been devised (see Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] On the other hand, it is desired to realize image recognition and the like using a convolutional neural network even in embedded devices such as IoT devices. In an embedded device, it is difficult to incorporate a large-scale dedicated circuit described in Patent Document 1 or the like. Further, in an embedded device with limited hardware resources such as a CPU and a memory, it is difficult to realize sufficient operation performance of a convolutional neural network only by software.

[0005] In view of the above circumstances, an object of the present invention is to provide a neural network circuit and a method for controlling a neural network circuit that can operate a neural network circuit that can be incorporated into an embedded device such as an IoT device with high performance.

Means for Solving the Problems

[0006] To solve the above problems, this invention proposes the following means. A neural network circuit according to a first aspect of the present invention comprises a convolution circuit that performs a convolution operation on input data, and a quantization circuit that performs a quantization operation on the output data of the convolution operation of the convolution circuit, wherein the convolution circuit enables clock gating of a first clock supplied to at least a portion of the convolution circuit when waiting to perform the convolution operation. [Effects of the Invention]

[0007] The neural network circuit and control method of the present invention enable high-performance operation of a neural network circuit that can be embedded in embedded devices such as IoT devices. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram of a convolutional neural network. [Figure 2] This diagram illustrates the convolution operation performed by a convolutional layer. [Figure 3] This is a diagram illustrating the data expansion in a convolution operation. [Figure 4] This figure shows the overall configuration of the neural network circuit according to the first embodiment. [Figure 5] This is a timing chart showing an example of the operation of the neural network circuit. [Figure 6] This is a timing chart showing other operational examples of the same neural network circuit. [Figure 7] This is an internal block diagram of the DMAC (Digital Media Controller) neural network circuit. [Figure 8] This is a state transition diagram of the control circuit of the DMAC. [Figure 9] This is a timing chart showing the operation of the clock control unit 39. [Figure 10] It is an internal block diagram of the convolution operation circuit of the same neural network circuit. [Figure 11] It is an internal block diagram of the multiplier of the same convolution operation circuit. [Figure 12] It is an internal block diagram of the product-sum operation unit of the same multiplier. [Figure 13] It is an internal block diagram of the accumulator circuit of the same convolution operation circuit. [Figure 14] It is an internal block diagram of the accumulator unit of the same accumulator circuit. [Figure 15] It is an internal block diagram of the quantization operation circuit of the same neural network circuit. [Figure 16] It is an internal block diagram of the vector operation circuit and the quantization circuit of the same quantization operation circuit. [Figure 17] It is a block diagram of the operation unit. [Figure 18] It is an internal block diagram of the vector quantization unit of the same quantization circuit. [Figure 19] It is a diagram for explaining the control of the same neural network circuit by a semaphore. [Figure 20] It is a timing chart of the first data flow. [Figure 21] It is a timing chart of the second data flow. [Figure 22] It is a diagram for explaining the convolution operation execution instruction. [Figure 23] It is a diagram showing a specific example of the convolution operation instruction. [Figure 24] It is a diagram for explaining the quantization operation execution instruction. [Figure 25] It is a diagram for explaining the DMA transfer execution instruction. [Figure 26] It is a diagram showing the convolution operation instruction of the neural network circuit according to the second embodiment. [Figure 27] It is a diagram showing a specific example of the convolution operation instruction of the same neural network circuit.

Embodiments for Carrying Out the Invention

[0009] (First Embodiment) A first embodiment of the present invention will be described with reference to Figures 1 to 25. Figure 1 shows a convolutional neural network 200 (hereinafter referred to as "CNN200"). The operations performed by the neural network circuit 100 (hereinafter referred to as "NN circuit 100") according to the first embodiment are at least a part of the trained CNN200 used during inference.

[0010] [CNN200] CNN200 is a multilayer network that includes a convolutional layer 210 for performing convolution operations, a quantization layer 220 for performing quantization operations, and an output layer 230. In at least a portion of CNN200, the convolutional layer 210 and the quantization layer 220 are alternately connected. CNN200 is a model widely used for image recognition and video recognition. CNN200 may further have layers with other functions, such as fully connected layers.

[0011] Figure 2 illustrates the convolution operation performed by the convolutional layer 210. The convolutional layer 210 performs a convolution operation on the input data a using the weights w. The convolutional layer 210 also performs a sum-of-products operation with the input data a and the weights w as inputs.

[0012] The input data a (also called activation data or feature map) to the convolutional layer 210 is multidimensional data such as image data. In this embodiment, the input data a is a 3D tensor consisting of elements (x, y, c). The convolutional layer 210 of the CNN200 performs a convolution operation on the low-bit input data a. In this embodiment, the elements of the input data a are 2-bit unsigned integers (0, 1, 2, 3). The elements of the input data a may also be, for example, 4-bit or 8-bit unsigned integers.

[0013] If the input data to CNN200 is in a different format from the input data a to the convolutional layer 210, such as a 32-bit floating-point type, CNN200 may have an additional input layer before the convolutional layer 210 to perform type conversion and quantization.

[0014] The weights w (also called filters or kernels) of the convolutional layer 210 are multidimensional data having elements that are learnable parameters. In this embodiment, the weights w are a 4-dimensional tensor consisting of elements (i, j, c, d). The weights w have d 3-dimensional tensors (hereinafter referred to as "weights w") consisting of elements (i, j, c). The weights w in a trained CNN200 are trained data. The convolutional layer 210 of the CNN200 performs convolution operations using low-bit weights w. In this embodiment, the elements of the weights w are 1-bit signed integers (0, 1), where the value "0" represents +1 and the value "1" represents -1.

[0015] The convolutional layer 210 performs the convolution operation shown in Equation 1 and outputs output data f. In Equation 1, s represents the stride. In Figure 2, the region shown by the dotted line represents one of the regions ao (hereinafter referred to as "application region ao") to which the weight wo is applied to the input data a. The elements of the application region ao are represented by (x+i, y+j, c).

[0016]

number

[0017] The quantization layer 220 performs quantization on the output of the convolution operation output by the convolution layer 210. The quantization layer 220 includes a pooling layer 221, a batch normalization layer 222, an activation function layer 223, and a quantization layer 224.

[0018] The pooling layer 221 compresses the output data f of the convolution operation output by the convolution layer 210 by performing operations such as average pooling (Equation 2) and MAX pooling (Equation 3). In Equations 2 and 3, u represents the input tensor, v represents the output tensor, and T represents the size of the pooling region. In Equation 3, max is a function that outputs the maximum value of u for the combination of i and j included in T.

[0019]

number

[0020]

number

[0021] The Batch Normalization layer 222 normalizes the data distribution of the output data from the quantization layer 220 and the pooling layer 221 using an operation such as that shown in Equation 4. In Equation 4, u represents the input tensor, v represents the output tensor, α represents the scale, and β represents the bias. In the trained CNN200, α and β are trained constant vectors.

[0022]

number

[0023] The activation function layer 223 performs activation function operations such as ReLU (Equation 5) on the outputs of the quantization operation layer 220, the pooling layer 221, and the Batch Normalization layer 222. In Equation 5, u is the input tensor and v is the output tensor. In Equation 5, max is a function that outputs the largest number among the arguments.

[0024]

number

[0025] The quantization layer 224 performs quantization on the outputs of the pooling layer 221 and the activation function layer 223 based on the quantization parameters, for example, as shown in Equation 6. The quantization shown in Equation 6 reduces the input tensor u to 2 bits. In Equation 6, q(c) is a vector of quantization parameters. In a trained CNN200, q(c) is a trained constant vector. The inequality sign "≦" in Equation 6 may also be "<".

[0026]

number

[0027] The output layer 230 is the layer that outputs the results of CNN200 using an identity function, a softmax function, or the like. The layer preceding the output layer 230 may be a convolutional layer 210 or a quantization operation layer 220.

[0028] In CNN200, the output data of the quantized quantization layer 224 is input to the convolutional layer 210, resulting in a lower convolutional computation load on the convolutional layer 210 compared to other convolutional neural networks that do not perform quantization.

[0029] [Partitioning of Convolution Operations] The NN circuit 100 performs the convolution operation (Equation 1) of the convolution layer 210 by dividing the input data into subtensors. The method of dividing into subtensors and the number of divisions are not particularly limited. A subtensor is formed, for example, by dividing the input data a(x+i,y+j,c) into a(x+i,y+j,co). The NN circuit 100 can also perform the convolution operation (Equation 1) of the convolution layer 210 without dividing the input data.

[0030] In the input data partitioning for the convolution operation, the variable c in Equation 1 is partitioned into blocks of size Bc, as shown in Equation 7. Similarly, the variable d in Equation 1 is partitioned into blocks of size Bd, as shown in Equation 8. In Equation 7, co is the offset and ci is the index from 0 to (Bc-1). In Equation 8, do is the offset and di is the index from 0 to (Bd-1). Note that sizes Bc and Bd may be the same.

[0031]

number

[0032]

number

[0033] In Equation 1, the input data a(x+i, y+j, c) is divided along the c-axis by size Bc, and the divided input data is represented as a(x+i, y+j, co). In the following explanation, the divided input data a will also be referred to as "divided input data a".

[0034] The weight w(i,j,c,d) in Equation 1 is divided by size Bc in the c-axis direction and size Bd in the d-axis direction, and is represented by the divided weight w(i,j,co,do). In the following explanation, the divided weight w will also be called the "divided weight w".

[0035] The output data f(x,y,do) divided by size Bd can be obtained using Equation 9. By combining the divided output data f(x,y,do), the final output data f(x,y,d) can be calculated.

[0036]

number

[0037] [Data expansion in convolution operations] The NN circuit 100 performs a convolution operation by expanding the input data a and the weights w in the convolution operation of the convolutional layer 210.

[0038] FIG. 3 is a diagram for explaining the expansion of data in the convolution operation. The divided input data a(x + i, y + j, co) is expanded into vector data having Bc elements. The elements of the divided input data a are indexed by ci (0 ≤ ci < Bc). In the following description, the divided input data a expanded into vector data for each i and j is also referred to as an "input vector A". The input vector A has elements from the divided input data a(x + i, y + j, co × Bc) to the divided input data a(x + i, y + j, co × Bc + (Bc - 1)).

[0039] The divided weight w(i, j, co, do) is expanded into matrix data having Bc × Bd elements. The elements of the divided weight w expanded into matrix data are indexed by ci and di (0 ≤ di < Bd). In the following description, the divided weight w expanded into matrix data for each i and j is also referred to as a "weight matrix W". The weight matrix W has elements from the divided weight w(i, j, co × Bc, do × Bd) to the divided weight w(i, j, co × Bc + (Bc - 1), do × Bd + (Bd - 1)).

[0040] By multiplying the input vector A and the weight matrix W, vector data is calculated. By shaping the vector data calculated for each i, j, and co into a three-dimensional tensor, the output data f(x, y, do) can be obtained. By performing such data expansion, the convolution operation of the convolutional layer 210 can be implemented by multiplying vector data and matrix data.

[0041] [NN circuit 100] FIG. 4 is a diagram showing the overall configuration of the NN circuit 100 according to the present embodiment. The NN circuit 100 comprises a first memory 1, a second memory 2, a DMA controller 3 (hereinafter also referred to as "DMAC3"), a convolution operation circuit 4, a quantization operation circuit 5, and a controller 6. The NN circuit 100 is characterized in that the convolution operation circuit 4 and the quantization operation circuit 5 are formed in a loop via the first memory 1 and the second memory 2.

[0042] The first memory 1 is a rewritable memory, such as a volatile memory composed of SRAM (Static RAM). Data is written to and read from the first memory 1 via the DMAC 3 and controller 6. The first memory 1 is connected to the input port of the convolution arithmetic circuit 4, and the convolution arithmetic circuit 4 can read data from the first memory 1. The first memory 1 is also connected to the output port of the quantization arithmetic circuit 5, and the quantization arithmetic circuit 5 can write data to the first memory 1. The external host CPU 110 can input and output data to the NN circuit 100 by writing and reading data to and from the first memory 1.

[0043] The second memory 2 is a rewritable memory, such as a volatile memory composed of SRAM (Static RAM). Data is written to and read from the second memory 2 via the DMAC 3 and controller 6. The second memory 2 is connected to the input port of the quantization arithmetic circuit 5, and the quantization arithmetic circuit 5 can read data from the second memory 2. The second memory 2 is also connected to the output port of the convolution arithmetic circuit 4, and the convolution arithmetic circuit 4 can write data to the second memory 2. The external host CPU 110 can input and output data to the NN circuit 100 by writing and reading data to and from the second memory 2.

[0044] DMAC3 is connected to the external bus EB and performs data transfer between the external memory 120, such as DRAM, and the first memory 1. DMAC3 also performs data transfer between the external memory 120, such as DRAM, and the second memory 2. Furthermore, DMAC3 performs data transfer between the external memory 120, such as DRAM, and the convolution arithmetic circuit 4. Finally, DMAC3 performs data transfer between the external memory 120, such as DRAM, and the quantization arithmetic circuit 5.

[0045] The convolution operation circuit 4 is a circuit that performs convolution operations in the convolution layer 210 of the trained CNN200. The convolution operation circuit 4 reads the input data a stored in the first memory 1 and performs a convolution operation on the input data a. The convolution operation circuit 4 writes the output data f of the convolution operation (hereinafter also referred to as "convolution operation output data") to the second memory 2.

[0046] The quantization circuit 5 is a circuit that performs at least a portion of the quantization operations in the quantization layer 220 of the trained CNN200. The quantization circuit 5 reads the output data f of the convolution operation stored in the second memory 2 and performs a quantization operation (an operation that includes at least quantization, among pooling, batch normalization, activation function, and quantization) on the output data f of the convolution operation. The quantization circuit 5 writes the output data of the quantization operation (hereinafter also referred to as "quantization operation output data") to the first memory 1.

[0047] Controller 6 is connected to the external bus EB and operates as both a master and a slave to the external bus EB. Controller 6 has register 61 and IFU 62.

[0048] Register 61 contains parameter registers and state registers. The parameter registers control the operation of the NN circuit 100. The state registers indicate the state of the NN circuit 100, including the semaphore S. The external host CPU 110 can access register 61 via the controller 6.

[0049] The Instruction Fetch Unit (IFU) 62 reads instruction commands for the DMAC 3, convolution circuit 4, and quantization circuit 5 via the external bus EB based on instructions from the external host CPU 110. The IFU 62 also transfers the read instruction commands to the corresponding DMAC 3, convolution circuit 4, and quantization circuit 5.

[0050] Controller 6 is connected to the first memory 1, the second memory 2, the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via dedicated wiring (not shown) connected to the internal bus IB (see Figure 4) and IFU 62. The external host CPU 110 can access each block via Controller 6. For example, the external host CPU 110 can issue instructions to the DMAC 3, the convolution circuit 4, and the quantization circuit 5 via Controller 6. In addition, the DMAC 3, the convolution circuit 4, and the quantization circuit 5 can update the state registers (including semaphores S) of Controller 6 via the internal bus IB. The state registers (including semaphores S) may be configured to be updated via dedicated wiring connected to the DMAC 3, the convolution circuit 4, and the quantization circuit 5.

[0051] Since the NN circuit 100 has a first memory 1 and a second memory 2, etc., it can reduce the number of data transfers of duplicate data when transferring data from an external memory 120 such as DRAM by the DMAC 3. This can significantly reduce the power consumption generated by memory access.

[0052] [Example of operation of NN circuit 100 1] Figure 5 is a timing chart showing an example of the operation of the NN circuit 100. DMAC3 stores the input data a from layer 1 into the first memory 1. DMAC3 may also divide the input data a from layer 1 and transfer it to the first memory 1 in accordance with the order of the convolution operations performed by the convolution operation circuit 4.

[0053] The convolution circuit 4 reads the input data a of layer 1 stored in the first memory 1. The convolution circuit 4 performs the convolution operation of layer 1 shown in Figure 1 on the input data a of layer 1. The output data f of the convolution operation of layer 1 is stored in the second memory 2.

[0054] The quantization circuit 5 reads the output data f of Layer 1 stored in the second memory 2. The quantization circuit 5 performs the Layer 2 quantization operation on the output data f of Layer 1. The output data of the Layer 2 quantization operation is stored in the first memory 1.

[0055] The convolution circuit 4 reads the output data of the quantization operation of layer 2 stored in the first memory 1. The convolution circuit 4 uses the output data of the quantization operation of layer 2 as input data a and performs the convolution operation of layer 3. The output data f of the convolution operation of layer 3 is stored in the second memory 2.

[0056] The convolution circuit 4 reads the output data of the quantization operation of layer 2M-2 (where M is a natural number) stored in the first memory 1. The convolution circuit 4 uses the output data of the quantization operation of layer 2M-2 as input data a and performs the convolution operation of layer 2M-1. The output data f of the convolution operation of layer 2M-1 is stored in the second memory 2.

[0057] The quantization circuit 5 reads the output data f of layer 2M-1 stored in the second memory 2. The quantization circuit 5 performs a quantization operation on the output data f of layer 2M-1. The output data of the quantization operation of layer 2M is stored in the first memory 1.

[0058] The convolution circuit 4 reads the output data of the quantization operation of layer 2M stored in the first memory 1. The convolution circuit 4 uses the output data of the quantization operation of layer 2M as input data a and performs a convolution operation on layer 2M+1. The output data f of the convolution operation of layer 2M+1 is stored in the second memory 2.

[0059] The convolution circuit 4 and the quantization circuit 5 perform operations alternately, advancing the CNN200 operation shown in Figure 1. In the NN circuit 100, the convolution circuit 4 performs the convolution operation of layer 2M-1 and layer 2M+1 using time-division multiplexing. In addition, the NN circuit 100 performs the quantization operation of layer 2M-2 and layer 2M using time-division multiplexing. Therefore, the NN circuit 100 has a significantly smaller circuit size compared to the case where separate convolution circuits 4 and quantization circuits 5 are implemented for each layer.

[0060] The NN circuit 100 performs calculations for the CNN200, which has a multilayer structure of multiple layers, using a loop-shaped circuit. The loop-shaped circuit configuration of the NN circuit 100 allows for efficient use of hardware resources. In order to form the circuit in a loop, the parameters of the convolutional operation circuit 4 and quantization operation circuit 5, which change in each layer, are updated as needed.

[0061] If the CNN200 calculations include operations that cannot be performed by the NN circuit 100, the NN circuit 100 transfers intermediate data to an external computing device such as an external host CPU 110. After the external computing device performs calculations on the intermediate data, the results of the calculations performed by the external computing device are input into the first memory 1 and the second memory 2. The NN circuit 100 then resumes calculations on the results of the calculations performed by the external computing device.

[0062] [Example of operation of NN circuit 100 2] Figure 6 is a timing chart showing another example of operation of the NN circuit 100. The NN circuit 100 may divide the input data a into subtensors and perform operations on the subtensors using time-division multiplexing. The method of dividing into subtensors and the number of divisions are not particularly limited.

[0063] Figure 6 shows an example of operation when input data a is decomposed into two subtensors. The decomposed subtensors are denoted as "first subtensor a1" and "second subtensor a2". For example, the convolution operation of layer 2M-1 is decomposed into a convolution operation corresponding to the first subtensor a1 (denoted as "layer 2M-1(a1)" in Figure 6) and a convolution operation corresponding to the second subtensor a2 (denoted as "layer 2M-1(a2)" in Figure 6).

[0064] The convolution and quantization operations corresponding to the first subtensor a1 and the convolution and quantization operations corresponding to the second subtensor a2 can be performed independently, as shown in Figure 6.

[0065] The convolution circuit 4 performs the convolution operation on layer 2M-1 corresponding to the first subtensor a1 (shown as layer 2M-1(a1) in Figure 6). Subsequently, the convolution circuit 4 performs the convolution operation on layer 2M-1 corresponding to the second subtensor a2 (shown as layer 2M-1(a2) in Figure 6). The quantization circuit 5 also performs the quantization operation on layer 2M corresponding to the first subtensor a1 (shown as layer 2M(a1) in Figure 6). In this way, the NN circuit 100 can perform the convolution operation on layer 2M-1 corresponding to the second subtensor a2 and the quantization operation on layer 2M corresponding to the first subtensor a1 in parallel.

[0066] Next, the convolution circuit 4 performs the convolution operation on layer 2M+1 corresponding to the first subtensor a1 (indicated as layer 2M+1(a1) in Figure 6). The quantization circuit 5 then performs the quantization operation on layer 2M corresponding to the second subtensor a2 (indicated as layer 2M(a2) in Figure 6). Thus, the NN circuit 100 can perform the convolution operation on layer 2M+1 corresponding to the first subtensor a1 and the quantization operation on layer 2M corresponding to the second subtensor a2 in parallel.

[0067] The convolution and quantization operations corresponding to the first subtensor a1 and the convolution and quantization operations corresponding to the second subtensor a2 can be performed independently. Therefore, the NN circuit 100 may, for example, perform the convolution operation of layer 2M-1 corresponding to the first subtensor a1 and the quantization operation of layer 2M+2 corresponding to the second subtensor a2 in parallel. In other words, the convolution and quantization operations performed in parallel by the NN circuit 100 are not limited to operations on consecutive layers.

[0068] By dividing the input data a into subtensors, the NN circuit 100 can operate the convolution circuit 4 and the quantization circuit 5 in parallel. As a result, the waiting time for the convolution circuit 4 and the quantization circuit 5 is reduced, improving the computational efficiency of the NN circuit 100. In the example shown in Figure 6, the number of divisions was 2, but similarly, the NN circuit 100 can operate the convolution circuit 4 and the quantization circuit 5 in parallel even when the number of divisions is greater than 2.

[0069] For example, if the input data a is divided into "first subtensor a1", "second subtensor a2", and "third subtensor a3", the NN circuit 100 may perform the convolution operation of layer 2M-1 corresponding to the second subtensor a2 and the quantization operation of layer 2M corresponding to the third subtensor a3 in parallel. The order of operations is appropriately changed depending on the storage status of the input data a in the first memory 1 and the second memory 2.

[0070] As an example of how to perform operations on subtensors, we showed an example (Method 1) in which operations on subtensors in the same layer are performed in the convolution circuit 4 or quantization circuit 5, and then operations on subtensors in the next layer are performed. For example, as shown in Figure 6, in the convolution circuit 4, after performing the convolution operation on layer 2M-1 corresponding to the first subtensor a1 and the second subtensor a2 (operations shown as layer 2M-1(a1) and layer 2M-1(a2) in Figure 6), the convolution operation on layer 2M+1 corresponding to the first subtensor a1 and the second subtensor a2 (operations shown as layer 2M+1(a1) and layer 2M+1(a2) in Figure 6) is performed.

[0071] However, the method of operation on subtensors is not limited to this. The method of operation on subtensors may also be to perform operations on some subtensors in multiple layers and then perform operations on the remaining subtensors (Method 2). For example, in the convolution operation circuit 4, after performing a convolution operation on layer 2M-1 corresponding to the first subtensor a1 and layer 2M+1 corresponding to the first subtensor a1, a convolution operation on layer 2M-1 corresponding to the second subtensor a2 and layer 2M+1 corresponding to the second subtensor a2 may be performed.

[0072] Furthermore, the method for performing operations on a subtensor may also be a combination of Method 1 and Method 2. However, when using Method 2, it is necessary to perform operations according to the dependencies on the order of operations of the subtensor.

[0073] Next, we will explain in detail each component of the NN circuit 100.

[0074] [DMAC3] Figure 7 is an internal block diagram of DMAC3. The DMAC3 comprises a data transfer circuit 31, a state controller 32, and a clock control unit 39. The DMAC3 has a dedicated state controller 32 for the data transfer circuit 31, and when an instruction command is input, it can perform DMA data transfer without requiring an external controller.

[0075] The data transfer circuit 31 is connected to the external bus EB and performs DMA data transfer between the external memory 120, such as a DRAM, and the first memory 1. The data transfer circuit 31 also performs DMA data transfer between the external memory 120, such as a DRAM, and the second memory 2. Furthermore, the data transfer circuit 31 performs data transfer between the external memory 120, such as a DRAM, and the convolution arithmetic circuit 4. Finally, the data transfer circuit 31 performs data transfer between the external memory 120, such as a DRAM, and the quantization arithmetic circuit 5. The number of DMA channels in the data transfer circuit 31 is not limited. For example, the first memory 1 and the second memory 2 may each have their own dedicated DMA channel.

[0076] The state controller 32 controls the state of the data transfer circuit 31. The state controller 32 is also connected to the controller 6 via dedicated wiring (not shown) connected to the internal bus IB (see Figure 4) and the IFU 62. The state controller 32 includes an instruction queue 33 and a control circuit 34.

[0077] The instruction queue 33 is a queue that stores instruction commands C3 for DMAC3, and is composed of, for example, a FIFO memory. One or more instruction commands C3 are written to the instruction queue 33 via the internal bus IB or IFU62.

[0078] The control circuit 34 is a state machine that decodes instruction command C3 and sequentially controls the data transfer circuit 31 based on instruction command C3. The control circuit 34 may be implemented by logic circuits or by a CPU controlled by software.

[0079] Figure 8 is a state transition diagram of the control circuit 34. When the instruction command C3 is input to the instruction queue 33 (not empty), the control circuit 34 transitions from the idle state ST1 to the decode state ST2.

[0080] In the decode state ST2, the control circuit 34 decodes the instruction command C3 output from the instruction queue 33. The control circuit 34 also reads the semaphore S stored in the controller 6's register 61 and determines whether the operation of the data transfer circuit 31 instructed by the instruction command C3 is possible. If it is not possible (Not ready), the control circuit 34 transitions from the decode state ST2 to the wait state ST4. If it is possible (ready), the control circuit 34 transitions from the decode state ST2 to the execution state ST3.

[0081] In the wait state ST4, the control circuit 34 waits (Wait) until the operation of the data transfer circuit 31 instructed by the command C3 becomes possible. When it becomes ready, the control circuit 34 transitions from the wait state ST4 to the execution state ST3.

[0082] In the execution state ST3, the control circuit 34 controls the data transfer circuit 31 to perform the operation instructed by the instruction command C3. When the operation of the data transfer circuit 31 is complete, the control circuit 34 removes the completed instruction command C3 from the instruction queue 33 and updates the semaphore S stored in the controller 6's register 61. If there is an instruction in the instruction queue 33 (Not empty), the control circuit 34 transitions from the execution state ST3 to the decode state ST2. If there is no instruction in the instruction queue 33 (empty), the control circuit 34 transitions from the execution state ST3 to the idle state ST1.

[0083] Figure 9 is a timing chart showing the operation of the clock control unit 39. The clock control unit 39 generates a gated clock (third clock) GC3 from the clock CK supplied to the NN circuit 100 based on the clock enable signal CE3. When the clock enable signal CE3 is negated and set to disabled (Low in Figure 9), the toggling of the gated clock GC3 is stopped. When the clock enable signal CE3 is asserted and set to enabled (High in Figure 9), the toggling of the gated clock GC3 is started. The gated clock GC3 generation circuit is a circuit appropriately selected from known clock gating circuits.

[0084] The clock enable signal CE3 is controlled by the state controller 32. When the state controller 32 transitions from the decode state ST2 to the wait state ST4, the control circuit 34 negates the clock enable signal CE3 and sets it to disabled. When the control circuit 34 is in the wait state ST4, the gated clock GC3 is stopped toggling. When the state controller 32 transitions from the wait state ST4 to the execution state ST3, the control circuit 34 asserts the clock enable signal CE3 and sets it to enabled. When the control circuit 34 is in the execution state ST3, the gated clock GC3 is restarted toggling.

[0085] The generated gated clock GC3 is output to a portion of the state controller 32 and the data transfer circuit 31, as shown in Figure 7, and is used as the operating clock.

[0086] [Convolution arithmetic circuit 4] Figure 10 is an internal block diagram of the convolution arithmetic circuit 4. The convolution arithmetic circuit 4 includes a weight memory 41, a multiplier 42, an accumulator circuit 43, a state controller 44, and a clock control unit 49. The convolution arithmetic circuit 4 has a dedicated state controller 44 for the multiplier 42 and the accumulator circuit 43, and when an instruction command is input, it can perform a convolution operation without requiring an external controller.

[0087] The weight memory 41 is a memory that stores the weights w used in the convolution operation, and is a rewritable memory such as a volatile memory composed of SRAM (Static RAM), for example. The DMAC3 writes the weights w required for the convolution operation to the weight memory 41 via DMA transfer.

[0088] Figure 11 is an internal block diagram of the multiplier 42. The multiplier 42 multiplies the input vector A and the weight matrix W. As described above, the input vector A is a vector data with Bc elements obtained by expanding the partitioned input data a(x+i, y+j, co) for each i and j. The weight matrix W is a matrix data with Bc × Bd elements obtained by expanding the partitioned weights w(i,j,co, do) for each i and j. The multiplier 42 has Bc × Bd multiply-accumulate units 47 and can perform the multiplication of the input vector A and the weight matrix W in parallel.

[0089] The multiplier 42 reads the input vector A and weight matrix W required for multiplication from the first memory 1 and weight memory 41, and performs the multiplication. The multiplier 42 outputs the result of Bd sum-of-accumulate operations O(di).

[0090] Figure 12 is an internal block diagram of the sum-of-accumulate unit 47. The multiply-accumulate unit 47 multiplies the input vector A's element A(ci) by the weight matrix W's element W(ci,di). The multiply-accumulate unit 47 then adds the result of this multiplication to the result S(ci,di) of another multiply-accumulate unit 47. The multiply-accumulate unit 47 outputs the summation result S(ci+1,di). Element A(ci) is a 2-bit unsigned integer (0,1,2,3). Element W(ci,di) is a 1-bit signed integer (0,1), where the value "0" represents +1 and the value "1" represents -1.

[0091] The multiply-accumulate unit 47 includes an inverter 47a, a selector 47b, and an adder 47c. The multiply-accumulate unit 47 performs multiplication using only the inverter 47a and selector 47b, without using a multiplier. Selector 47b selects the input of element A(ci) when element W(ci,di) is "0". Selector 47b selects the complement of element A(ci) inverted by the inverter when element W(ci,di) is "1". Element W(ci,di) is also input to the carry-in of adder 47c. Adder 47c outputs the value obtained by adding element A(ci) to S(ci,di) when element W(ci,di) is "0". Adder 47c outputs the value obtained by subtracting element A(ci) from S(ci,di) when W(ci,di) is "1".

[0092] Figure 13 is an internal block diagram of the accumulator circuit 43. The accumulator circuit 43 accumulates the sum-of-accumulate result O(di) of the multiplier 42 in the second memory 2. The accumulator circuit 43 has Bd accumulator units 48 and can accumulate Bd sum-of-accumulate results O(di) in the second memory 2 in parallel.

[0093] Figure 13 is an internal block diagram of the accumulator unit 48. The accumulator unit 48 includes an adder 48a and a mask unit 48b. The adder 48a adds the elements O(di) of the sum-of-accumulate result O to the partial sum, which is an intermediate result of the convolution operation shown in Equation 1 and stored in the second memory 2. The addition result is 16 bits per element. The addition result is not limited to 16 bits per element, but may be, for example, 15 bits or 17 bits per element.

[0094] The adder 48a writes the addition result to the same address in the second memory 2. The mask unit 48b, when the initialization signal clear is asserted, masks the output from the second memory 2 and sets the element O(di) to zero for addition. The initialization signal clear is asserted when the partial sum of the intermediate results is not stored in the second memory 2.

[0095] Once the convolution operation by the multiplier 42 and the accumulator circuit 43 is complete, the output data f(x,y,do) is stored in the second memory.

[0096] The state controller 44 controls the states of the multiplier 42 and the accumulator circuit 43. The state controller 44 is also connected to the controller 6 via dedicated wiring (not shown) connected to the internal bus IB (see Figure 4) and the IFU 62. The state controller 44 has an instruction queue 45 and a control circuit 46.

[0097] The instruction queue 45 is a queue that stores instruction commands C4 for the convolution arithmetic circuit 4, and is composed of, for example, a FIFO memory. Instruction commands C4 are written to the instruction queue 45 via the internal bus IB or IFU62.

[0098] The control circuit 46 is a state machine that decodes the instruction command C4 and controls the multiplier 42 and the accumulator circuit 43 based on the instruction command C4. The control circuit 46 has a similar configuration to the control circuit 34 of the state controller 32 of the DMAC3.

[0099] The clock control unit 49 generates a gated clock (first clock) GC4 from the clock CK supplied to the NN circuit 100 based on the clock enable signal CE4. As shown in Figure 9, when the clock enable signal CE4 is negated and set to disabled, the toggling of the gated clock GC4 is stopped. When the clock enable signal CE4 is asserted and set to enabled, the toggling of the gated clock GC4 is started. The clock control unit 49 has the same configuration as the clock control unit 39 of the DMAC3.

[0100] The clock enable signal CE4 is controlled by the state controller 44. When the state controller 44 transitions from the decode state ST2 to the wait state ST4, the control circuit 46 negates the clock enable signal CE4 and sets it to disabled. When the control circuit 46 is in the wait state ST4, the gated clock GC4 is stopped from toggling. When the state controller 44 transitions from the wait state ST4 to the execution state ST3, the control circuit 46 asserts the clock enable signal CE4 and sets it to enabled. When the control circuit 46 is in the execution state ST3, the gated clock GC4 is restarted from toggling.

[0101] The generated gated clock GC4 is output to a portion of the state controller 44, the weight memory 41, the multiplier 42, and the accumulator circuit 43, as shown in Figure 10, and is used as the operating clock.

[0102] [Quantization operation circuit 5] Figure 15 is an internal block diagram of the quantization operation circuit 5. The quantization operation circuit 5 includes a quantization parameter memory 51, a vector operation circuit 52, a quantization circuit 53, a state controller 54, and a clock control unit 59. The quantization operation circuit 5 has a dedicated state controller 54 for the vector operation circuit 52 and the quantization circuit 53, and when an instruction command is input, it can perform quantization operations without requiring an external controller.

[0103] The quantization parameter memory 51 is a memory that stores the quantization parameters q used in quantization calculations, and is a rewritable memory such as a volatile memory composed of SRAM (Static RAM), for example. The DMAC3 writes the quantization parameters q required for quantization calculations to the quantization parameter memory 51 by DMA transfer.

[0104] Figure 16 is an internal block diagram of the vector operation circuit 52 and the quantization circuit 53. The vector arithmetic circuit 52 performs operations on the output data f(x,y,do) stored in the second memory 2. The vector arithmetic circuit 52 has Bd arithmetic units 57 and performs SIMD operations in parallel on the output data f(x,y,do).

[0105] Figure 17 is a block diagram of the arithmetic unit 57. The arithmetic unit 57 includes, for example, an ALU 57a, a first selector 57b, a second selector 57c, a register 57d, and a shifter 57e. The arithmetic unit 57 may further include other arithmetic units, etc., that are present in known general-purpose SIMD arithmetic circuits.

[0106] The vector operation circuit 52 combines the arithmetic units and other components of the operation unit 57 to perform at least one operation on the output data f(x,y,do) from among the operations of the pooling layer 221, the batch normalization layer 222, and the activation function layer 223 in the quantization operation layer 220.

[0107] The arithmetic unit 57 can add the data stored in register 57d and the element f(di) of the output data f(x,y,do) read from the second memory 2 using ALU 57a. The arithmetic unit 57 can store the addition result from ALU 57a in register 57d. The arithmetic unit 57 can initialize the addition result by inputting "0" to ALU 57a in place of the data stored in register 57d, depending on the selection of the first selector 57b. For example, if the pooling area is 2x2, the shifter 57e can output the average value of the addition result by right-shifting the output of ALU 57a by 2 bits. The vector arithmetic circuit 52 can perform the average pooling calculation shown in Equation 2 by repeating the above calculations by Bd arithmetic units 57.

[0108] The arithmetic unit 57 can compare the data stored in register 57d with the element f(di) of the output data f(x,y,do) read from the second memory 2 using the ALU 57a. The arithmetic unit 57 controls the second selector 57c according to the comparison result by ALU 57a to select the larger of the data stored in register 57d and element f(di). The arithmetic unit 57 can initialize the comparison target to the minimum value by inputting the minimum possible value of element f(di) to ALU 57a based on the selection of the first selector 57b. In this embodiment, since element f(di) is a 16-bit signed integer, the minimum possible value of element f(di) is "0x8000". The vector arithmetic circuit 52 can perform the MAX pooling calculation of equation 3 by repeating the above calculations by Bd arithmetic units 57. Note that in the MAX pooling calculation, the shifter 57e does not shift the output of the second selector 57c.

[0109] The arithmetic unit 57 can subtract the data stored in register 57d from the element f(di) of the output data f(x,y,do) read from the second memory 2 using ALU 57a. The shifter 57e can left-shift (i.e. multiply) or right-shift (i.e. divide) the output of ALU 57a. The vector arithmetic circuit 52 can perform the Batch Normalization operation of Equation 4 by repeating the above operations by Bd arithmetic units 57.

[0110] The arithmetic unit 57 can compare the element f(di) of the output data f(x,y,do) read from the second memory 2 with "0" selected by the first selector 57b using the ALU 57a. The arithmetic unit 57 can select and output either the element f(di) or the constant value "0" previously stored in register 57d, depending on the comparison result by the ALU 57a. The vector arithmetic circuit 52 can perform the ReLU operation of equation 5 by repeating the above operations by Bd arithmetic units 57.

[0111] The vector operation circuit 52 can perform mean pooling, MAX pooling, batch normalization, activation function calculations, and combinations thereof. Since the vector operation circuit 52 can perform general-purpose SIMD operations, it may also perform other operations necessary for the operations in the quantization operation layer 220. Furthermore, the vector operation circuit 52 may also perform operations other than those in the quantization operation layer 220.

[0112] Note that the quantization circuit 5 does not necessarily have a vector operation circuit 52. If the quantization circuit 5 does not have a vector operation circuit 52, the output data f(x,y,do) is input to the quantization circuit 53.

[0113] The quantization circuit 53 performs quantization on the output data of the vector operation circuit 52. As shown in Figure 16, the quantization circuit 53 has Bd quantization units 58 and performs operations in parallel on the output data of the vector operation circuit 52.

[0114] FIG. 18 is an internal block diagram of the quantization unit 58. The quantization unit 58 quantizes the elements in(di) of the output data of the vector operation circuit 52. The quantization unit 58 includes a comparator 58a and an encoder 58b. The quantization unit 58 performs the operation (Equation 6) of the quantization layer 224 in the quantization operation layer 220 on the output data (16 bits / element) of the vector operation circuit 52. The quantization unit 58 reads the necessary quantization parameters q(th0, th1, th2) from the quantization parameter memory 51, and the comparator 58a compares the input in(di) with the quantization parameters q. The quantization unit 58 quantizes the comparison result by the comparator 58a to 2 bits / element by the encoder 58b. Since α(c) and β(c) in Equation 4 are different parameters for each variable c, the quantization parameters q(th0, th1, th2) reflecting α(c) and β(c) are different parameters for each in(di).

[0115] The quantization unit 58 classifies the input in(di) into four regions (for example, in≦th0, th0<in≦th1, th1<in≦th2, th2<in) by comparing the input in(di) with three threshold values th0, th1, th2, encodes the classification result into 2 bits, and outputs it. The quantization unit 58 can also perform operations of Batch Normalization and activation functions together with quantization by setting the quantization parameters q(th0, th1, th2).

[0116] The quantization unit 58 can perform the operation of Batch Normalization shown in Equation 4 together with quantization by setting the threshold th0 as β(c) in Equation 4 and the threshold differences (th1−th0) and (th2−th1) as α(c) in Equation 4. By increasing (th1−th0) and (th2−th1), α(c) can be decreased. By decreasing (th1−th0) and (th2−th1), α(c) can be increased.

[0117] The quantization unit 58 can perform the ReLU operation of the activation function in addition to the quantization of the input in(di). For example, the quantization unit 58 saturates the output value in the regions where in(di) ≤ th0 and th2 < in(di). The quantization unit 58 can perform the operation of the activation function in combination with quantization by setting the quantization parameter q so that the output is non-linear.

[0118] The state controller 54 controls the states of the vector operation circuit 52 and the quantization circuit 53. Also, the state controller 54 is connected to the controller 6 via an internal bus IB (see FIG. 4) and a dedicated wiring (not shown) connected to the IFU 62. The state controller 54 has an instruction queue 55 and a control circuit 56.

[0119] The instruction queue 55 is a queue in which the instruction command C5 for the quantization operation circuit 5 is stored, and is composed of, for example, a FIFO memory. The instruction command C5 is written into the instruction queue 55 via the internal bus IB or the IFU 62.

[0120] The control circuit 56 is a state machine that decodes the instruction command C5 and controls the vector operation circuit 52 and the quantization circuit 53 based on the instruction command C5. The control circuit 56 has the same configuration as the control circuit 34 of the state controller 32 of the DMAC3.

[0121] The clock control unit 59 generates a gated clock (third clock) GC5 from the clock CK supplied to the NN circuit 100 based on the clock enable signal CE5. As shown in FIG. 9, when the clock enable signal CE5 is negated and set to invalid (Disable), the toggling of the gated clock GC5 stops. When the clock enable signal CE5 is asserted and set to valid (Enable), the toggling of the gated clock GC5 starts. The clock control unit 59 has the same configuration as the clock control unit 39 of the DMAC3.

[0122] The clock enable signal CE5 is controlled by the state controller 54. When the state controller 54 transitions from the decode state ST2 to the wait state ST4, the control circuit 56 negates the clock enable signal CE5 and sets it to disabled. When the control circuit 56 is in the wait state ST4, the gated clock GC5 is stopped toggling. When the state controller 54 transitions from the wait state ST4 to the execution state ST3, the control circuit 56 asserts the clock enable signal CE5 and sets it to enabled. When the control circuit 56 is in the execution state ST3, the gated clock GC5 is restarted toggling.

[0123] The generated gated clock GC5 is output to a portion of the state controller 54, the quantization parameter memory 51, the vector operation circuit 52, and the quantization circuit 53, as shown in Figure 15, and is used as the operating clock.

[0124] The quantization circuit 5 writes the quantization output data, which has Bd elements, to the first memory 1. A suitable relationship between Bd and Bc is shown in Equation 10. In Equation 10, n is an integer.

[0125]

number

[0126] [Controller 6] Controller 6 is connected to the external bus EB and operates as both a master and a slave to the external bus EB. Controller 6 has registers 61, which include parameter registers and state registers, and an IFU 62. The parameter registers are registers that control the operation of the NN circuit 100. The state registers are registers that indicate the state of the NN circuit 100, which includes semaphore S.

[0127] The controller 6 transfers instruction commands to the instruction queues of the DMAC 3, convolution circuit 4, and quantization circuit 5 in two ways. The first method is to transfer instruction commands from the external host CPU 110 to the controller 6 via the internal bus IB (see Figure 4). The second method is for the IFU 62 to read instruction commands from the external memory 120 and transfer the instruction commands via dedicated wiring (not shown) connected to the IFU 62.

[0128] [Semaphore S] Figure 19 illustrates the control of the NN circuit 100 by the semaphore S. Semaphore S comprises a first semaphore S1, a second semaphore S2, and a third semaphore S3. Semaphore S is decremented by a P operation and incremented by a V operation. The P and V operations performed by DMAC3, the convolution circuit 4, and the quantization circuit 5 update semaphore S held by controller 6 via the internal bus IB.

[0129] The first semaphore S1 is used to control the first data flow F1. The first data flow F1 is a data flow in which the DMAC3 (Producer) writes input data a to the first memory 1 and the convolution arithmetic circuit 4 (Consumer) reads input data a. The first semaphore S1 has a first write semaphore S1W and a first read semaphore S1R.

[0130] The second semaphore S2 is used to control the second data flow F2. The second data flow F2 is a data flow in which the convolution arithmetic circuit 4 (Producer) writes output data f to the second memory 2, and the quantization arithmetic circuit 5 (Consumer) reads out the output data f. The second semaphore S2 has a second write semaphore S2W and a second read semaphore S2R.

[0131] The third semaphore S3 is used to control the third data flow F3. The third data flow F3 is a data flow in which the quantization arithmetic circuit 5 (Producer) writes the quantization arithmetic output data to the first memory 1, and the convolution arithmetic circuit 4 (Consumer) reads the quantization arithmetic output data from the quantization arithmetic circuit 5. The third semaphore S3 has a third write semaphore S3W and a third read semaphore S3R.

[0132] [First Data Flow F1] Figure 20 is a timing chart for the first data flow F1. The first write semaphore S1W is a semaphore that restricts writes to the first memory 1 by the DMAC3 in the first data flow F1. The first write semaphore S1W indicates the number of memory areas in the first memory 1 that have already had data read and can be written to, out of the memory areas that can store data of a predetermined size, such as input vector A. If the first write semaphore S1W is "0", the DMAC3 cannot write to the first memory 1 in the first data flow F1 and will wait until the first write semaphore S1W becomes "1" or greater.

[0133] The first read semaphore S1R is a semaphore that restricts the reading from the first memory 1 by the convolution arithmetic circuit 4 in the first data flow F1. The first read semaphore S1R indicates the number of memory areas in the first memory 1 that have already been written to and are readable, out of the memory areas that can store data of a predetermined size, such as an input vector A. When the first read semaphore S1R is "0", the convolution arithmetic circuit 4 cannot read from the first memory 1 in the first data flow F1 and is made to wait until the first read semaphore S1R becomes "1" or greater.

[0134] DMAC3 initiates a DMA transfer when command C3 is stored in command queue 33. As shown in Figure 20, since the first write semaphore S1W is not "0", DMAC3 initiates a DMA transfer (DMA transfer 1). When initiating the DMA transfer, DMAC3 performs a P operation on the first write semaphore S1W. After the DMA transfer is complete, DMAC3 performs a V operation on the first read semaphore S1R.

[0135] The convolution arithmetic circuit 4 starts the convolution operation when the instruction command C4 is stored in the instruction queue 45. As shown in Figure 20, since the first read semaphore S1R is "0", the convolution arithmetic circuit 4 is made to wait until the first read semaphore S1R becomes "1" or greater (Wait in the decode state ST4). When the state controller 44 of the convolution arithmetic circuit 4 transitions from the decode state ST2 to the wait state ST4, the control circuit 46 negates the clock enable signal CE4 and sets it to disabled. As a result, the toggling of the gated clock GC4 supplied to a part of the state controller 44, the weight memory 41, the multiplier 42, and the accumulator circuit 43 is stopped. As a result, the dynamic power associated with the gated clock GC4 in the wait state ST4 can be reduced. Note that when the control circuit 46 is in the wait state ST4, the remaining part of the state controller 44 that requires operations such as data communication with other blocks is provided with the clock CK instead of the gated clock GC4 and continues to operate.

[0136] When the first read semaphore S1R becomes "1" due to a V operation by DMAC3, the convolution circuit 4 starts the convolution operation (convolution operation 1). When the state controller 44 transitions from the wait state ST4 to the execution state ST3, the control circuit 46 asserts the clock enable signal CE4 to enable it. As a result, the gated clock GC4 is toggled again. Also, when the convolution operation starts, the convolution circuit 4 performs a P operation on the first read semaphore S1R. After the convolution operation is completed, the convolution circuit 4 performs a V operation on the first write semaphore S1W.

[0137] In Figure 20, when DMAC3 initiates the DMA transfer labeled "DMA transfer 3," the first write semaphore S1W is "0," so DMAC3 waits until the first write semaphore S1W becomes "1" or greater (Wait in decode state ST4). When the DMAC3's state controller 32 transitions from decode state ST2 to wait state ST4, the control circuit 34 negates the clock enable signal CE3 and sets it to disabled. As a result, the toggling of the gated clock GC3 provided to part of the state controller 32 and the data transfer circuit 31 is stopped. Consequently, the dynamic power associated with the gated clock GC3 in wait state ST4 can be reduced. When the control circuit 34 is in wait state ST4, the remaining part of the state controller 32 that requires operations such as data communication with other blocks is provided with clock CK instead of gated clock GC3 and continues to operate.

[0138] When the V operation by the convolution arithmetic circuit 4 causes the first write semaphore S1W to become "1" or greater, the DMAC3 starts a DMA transfer. The control circuit 34 of the state controller 32 asserts the clock enable signal CE3 to enable it when transitioning from the wait state ST4 to the execution state ST3. As a result, the gated clock GC3 is toggled again.

[0139] The DMAC3 and the convolution circuit 4 can prevent access contention to the first memory 1 in the first data flow F1 by using semaphore S1. Furthermore, the DMAC3 and the convolution circuit 4 can operate independently in parallel while synchronizing data transfer in the first data flow F1 by using semaphore S1.

[0140] [Second Dataflow F2] Figure 21 is a timing chart for the second data flow F2. The second write semaphore S2W is a semaphore that restricts writing to the second memory 2 by the convolution arithmetic circuit 4 in the second data flow F2. The second write semaphore S2W indicates the number of memory areas in the second memory 2 that have already been read and can be written to, out of the memory areas that can store data of a predetermined size, such as output data f. When the second write semaphore S2W is "0", the convolution arithmetic circuit 4 cannot write to the second memory 2 in the second data flow F2 and is made to wait until the second write semaphore S2W becomes "1" or greater.

[0141] The second read semaphore S2R is a semaphore that restricts the reading from the second memory 2 by the quantization arithmetic circuit 5 in the second data flow F2. The second read semaphore S2R indicates the number of memory areas in the second memory 2 that have already been written to and are readable, out of the memory areas that can store data of a predetermined size, such as output data f. If the second read semaphore S2R is "0", the quantization arithmetic circuit 5 cannot read from the second memory 2 in the second data flow F2 and is made to wait until the second read semaphore S2R becomes "1" or greater.

[0142] As shown in Figure 21, the convolution circuit 4 performs a P operation on the second write semaphore S2W when starting the convolution operation. After the convolution operation is completed, the convolution circuit 4 performs a V operation on the second read semaphore S2R.

[0143] The quantization arithmetic circuit 5 starts quantization when the instruction command C5 is stored in the instruction queue 55. As shown in Figure 21, since the second read semaphore S2R is "0", the quantization arithmetic circuit 5 is made to wait until the second read semaphore S2R becomes "1" or greater (Wait in decode state ST4). When the state controller 54 of the quantization arithmetic circuit 5 transitions from decode state ST2 to wait state ST4, the control circuit 56 negates the clock enable signal CE5 and sets it to disabled. As a result, the toggling of the gated clock GC5 supplied to a part of the state controller 54, the quantization parameter memory 51, the vector arithmetic circuit 52, and the quantization circuit 53 is stopped. As a result, the dynamic power associated with the gated clock GC5 in wait state ST4 can be reduced. Note that when the control circuit 56 is in wait state ST4, the remaining part of the state controller 54 that requires operations such as data communication with other blocks is provided with clock CK instead of the gated clock GC5 and continues to operate.

[0144] When the second read semaphore S2R becomes "1" due to a V operation by the convolution arithmetic circuit 4, the quantization arithmetic circuit 5 starts the quantization operation (quantization operation 1). The control circuit 56 of the state controller 54 asserts the clock enable signal CE5 to enable it when transitioning from the wait state ST4 to the execution state ST3. As a result, the gated clock GC5 is toggled again. Also, when the quantization arithmetic circuit 5 starts the quantization operation, it performs a P operation on the second read semaphore S2R. After the quantization operation is completed, the quantization arithmetic circuit 5 performs a V operation on the second write semaphore S2W.

[0145] In Figure 21, when the quantization circuit 5 starts the quantization operation labeled "Quantization Operation 2," the second read semaphore S2R is "0," so the quantization circuit 5 is made to wait until the second read semaphore S2R becomes "1" or greater (Wait in Decode State ST4). When the quantization circuit 54's state controller 54 transitions from Decode State ST2 to Wait State ST4, the control circuit 56 negates the clock enable signal CE5 to disable it. When the second read semaphore S2R becomes "1" or greater due to the V operation by the convolution circuit 4, the quantization circuit 5 starts the quantization operation. When the state controller 54's control circuit 56 transitions from Wait State ST4 to Execution State ST3, it asserts the clock enable signal CE5 to enable it.

[0146] The convolution circuit 4 and the quantization circuit 5 can prevent access contention to the second memory 2 in the second data flow F2 by using semaphore S2. Furthermore, the convolution circuit 4 and the quantization circuit 5 can operate independently in parallel while synchronizing data transfer in the second data flow F2 by using semaphore S2.

[0147] [Third data flow F3] The third write semaphore S3W is a semaphore that restricts writing to the first memory 1 by the quantization arithmetic circuit 5 in the third data flow F3. The third write semaphore S3W indicates the number of memory areas in the first memory 1 that have already had data read out and can be written to, out of the memory areas that can store data of a predetermined size, such as the quantization arithmetic output data of the quantization arithmetic circuit 5. When the third write semaphore S3W is "0", the quantization arithmetic circuit 5 cannot write to the first memory 1 in the third data flow F3 and is made to wait until the third write semaphore S3W becomes "1" or greater.

[0148] The third read semaphore S3R is a semaphore that restricts the reading from the first memory 1 by the convolution arithmetic circuit 4 in the third data flow F3. The third read semaphore S3R indicates the number of memory areas in the first memory 1 that have already been written to and are readable, out of the memory areas that can store data of a predetermined size, such as the quantization arithmetic output data of the quantization arithmetic circuit 5. If the third read semaphore S3R is "0", the convolution arithmetic circuit 4 cannot read from the first memory 1 in the third data flow F3 and is made to wait until the third read semaphore S3R becomes "1" or greater.

[0149] The quantization circuit 5 and the convolution circuit 4 can prevent access contention to the first memory 1 in the third data flow F3 by using semaphore S3. Furthermore, the quantization circuit 5 and the convolution circuit 4 can operate independently in parallel while synchronizing data transfer in the third data flow F3 by using semaphore S3.

[0150] The first memory 1 is shared by the first data flow F1 and the third data flow F3. The NN circuit 100 can distinguish between the first data flow F1 and the third data flow F3 and synchronize data transfer by separately providing the first semaphore S1 and the third semaphore S3.

[0151] [Operation of the convolution circuit 4 (1)] When the convolution operation is performed, the convolution circuit 4 reads from the first memory 1 and writes to the second memory 2. In other words, the convolution circuit 4 is a Consumer in the first data flow F1 and a Producer in the second data flow F2. Therefore, when the convolution operation is started, the convolution circuit 4 performs a P operation on the first read semaphore S1R (see Figure 20) and a P operation on the second write semaphore S2W (see Figure 21). After the convolution operation is completed, the convolution circuit 4 performs a V operation on the first write semaphore S1W (see Figure 20) and a V operation on the second read semaphore S2R (see Figure 21).

[0152] The convolution circuit 4 waits until the first read semaphore S1R is "1" or greater and the second write semaphore S2W is "1" or greater before starting the convolution operation (Wait in the decode state ST4). During the period when the convolution circuit 4 is waiting due to the two semaphores, the gated clock GC4 is not toggled. As a result, the dynamic power associated with the gated clock GC4 in the wait state ST4 can be reduced.

[0153] The convolutional circuit 4 asserts the clock enable signal CE4 to enable it when the first read semaphore S1R is 1 or greater and the second write semaphore S2W is 1 or greater, and it transitions from the wait state ST4 to the execution state ST3. As a result, the gated clock GC4 is toggled again.

[0154] [Operation of Quantization Circuit 5] When the quantization operation is performed, the quantization circuit 5 reads from the second memory 2 and writes to the first memory 1. In other words, the quantization circuit 5 is a Consumer in the second data flow F2 and a Producer in the third data flow F3. Therefore, when the quantization operation is started, the quantization circuit 5 performs a P operation on the second read semaphore S2R and a P operation on the third write semaphore S3W. After the quantization operation is completed, the quantization circuit 5 performs a V operation on the second write semaphore S2W and a V operation on the third read semaphore S3R.

[0155] When the quantization arithmetic circuit 5 starts a quantization operation, it waits until the second read semaphore S2R is "1" or greater and the third write semaphore S3W is "1" or greater (Wait in the decode state ST4). During the period when the quantization arithmetic circuit 5 is waiting due to the two semaphores, the gated clock GC5 is not toggled. As a result, the dynamic power associated with the gated clock GC5 in the wait state ST4 can be reduced.

[0156] The quantization arithmetic circuit 5 asserts the clock enable signal CE5 to enable it when the second read semaphore S2R is 1 or greater and the third write semaphore S3W is 1 or greater, and the system transitions from the wait state ST4 to the execution state ST3. As a result, the gated clock GC5 is toggled again.

[0157] [Operation of the convolution circuit 4 (2)] The input data that the convolution circuit 4 reads from the first memory 1 may also be the data written by the quantization circuit 5 in the third data flow. In this case, the convolution circuit 4 is the Consumer in the third data flow F3 and the Producer in the second data flow F2. Therefore, when the convolution circuit 4 starts the convolution operation, it performs a P operation on the third read semaphore S3R and a P operation on the second write semaphore S2W. After the convolution operation is completed, the convolution circuit 4 performs a V operation on the third write semaphore S3W and a V operation on the second read semaphore S2R.

[0158] The convolution circuit 4 waits until the third read semaphore S3R is "1" or greater and the second write semaphore S2W is "1" or greater before starting the convolution operation (Wait in the decode state ST4). During the period when the convolution circuit 4 is waiting due to the two semaphores, the gated clock GC4 is not toggled. As a result, the dynamic power associated with the gated clock GC4 in the wait state ST4 can be reduced.

[0159] The convolutional circuit 4 asserts the clock enable signal CE4 to enable it when the first read semaphore S1R is 1 or greater and the second write semaphore S2W is 1 or greater, and it transitions from the wait state ST4 to the execution state ST3. As a result, the gated clock GC4 is toggled again.

[0160] [Convolution operation instruction] Figure 22 is a diagram illustrating the convolution operation instruction. The convolution operation execution instruction is one of the instruction commands C4 for the convolution operation circuit 4. The convolution operation execution instruction has an instruction field IF in which instructions for the convolution operation circuit 4 are stored, and a semaphore operation field SF in which operations on semaphore S are stored. The instruction field IF and the semaphore operation field SF are contained in a single instruction as the convolution operation execution instruction.

[0161] The instruction field IF of the convolution operation instruction is a field that stores instructions for the convolution operation circuit 4. For example, the instruction field IF stores commands to cause the multiplier 42 and the accumulator circuit 43 to perform the convolution operation, commands to control the clear signal of the accumulator circuit 43, and the sizes and memory addresses of the input vector A and the weight matrix W.

[0162] The semaphore operation field SF of the convolution operation instruction stores operations on the semaphore S associated with the instruction stored in the instruction field IF. The convolution operation circuit 4 is a Consumer that receives and consumes data from the other side in the first data flow F1 and the third data flow F3, and a Producer that transmits the data it has produced to the other side in the second data flow F2. Therefore, the associated semaphores S are the first semaphore S1, the second semaphore S2, and the third semaphore S3. As shown in Figure 22, the semaphore operation field SF of the convolution operation instruction includes operation fields for the first semaphore S1, the second semaphore S2, and the third semaphore S3.

[0163] The semaphore operation field SF has a P operation field and a V operation field for each semaphore. As shown in Figure 22, the semaphore operation field SF of a convolution operation instruction contains six operation fields. Each operation field in the semaphore operation field SF is 1 bit. Each operation field in the semaphore operation field SF may be multiple bits.

[0164] The first semaphore S1 and the third semaphore S3 for the first data flow F1 and the third data flow F3, for which the convolution arithmetic circuit 4 is the consumer, are provided with a P operation field for read semaphores (S1R, S3R) and a V operation field for write semaphores (S1W, S3W).

[0165] The second semaphore S2 for the second data flow F2, which is produced by the convolution arithmetic circuit 4, is provided with a P operation field for the write semaphore (S2W) and a V operation field for the read semaphore (S2R).

[0166] Figure 23 shows a specific example of a convolution operation instruction. The specific example shown in Figure 23 consists of four convolution operation instructions (hereinafter referred to as "instruction 1" to "instruction 4"), which divide the input data a(x+i,y+j,co) stored in the first memory 1 into four parts and cause the convolution operation circuit 4 to perform the convolution operation.

[0167] The state controller 44 of the convolution arithmetic circuit 4 transitions to the decode state ST2 and decodes instruction 1, which was the first instruction stored in the instruction queue 45 out of the four instructions (instructions 1 to 4).

[0168] If the P operation field is set to "1", the state controller 44 reads the semaphore S corresponding to the P operation field set to "1" from the controller 6 via the internal bus IB and determines whether the execution condition is met. The execution condition is that all semaphores S corresponding to the P operation field set to "1" are "1" or greater. In instruction 1, the P operation field for the first read semaphore S1R and the P operation field for the second write semaphore S2W are set to "1". Therefore, the state controller 44 reads the first read semaphore S1R and the second write semaphore S2W and determines whether the execution condition is met.

[0169] If the P operation field is set to "1", the state controller 44 waits until the semaphore S corresponding to the P operation field set to "1" is updated and the execution conditions are met. In the case of instruction 1, the state controller 44 waits until the semaphore S is updated and the execution conditions are met (Wait) unless the first read semaphore S1R is "1" or greater and the second write semaphore S2W is "1" or greater (Not Ready).

[0170] If the P operation field is set to "1", the state controller 44 transitions to execution state ST3 and performs a convolution operation based on the instruction field IF, provided that the execution conditions are met. For instruction 1, if the first read semaphore S1R is "1" or greater and the second write semaphore S2W is "1" or greater (Ready), the state controller 44 transitions to execution state ST3 and performs a convolution operation based on the instruction field IF.

[0171] If the P operation field is set to "1", the state controller 44 performs a P operation on the semaphore S corresponding to the P operation field set to "1" before performing the convolution operation. In the case of instruction 1, the state controller 44 performs a P operation on the first read semaphore S1R and the second write semaphore S2W before performing the convolution operation.

[0172] After executing instruction 1, the state controller 44 transitions to the decode state ST2 and decodes instruction 2. In instruction 2, none of the semaphore operation fields SF are set to "1". Therefore, the state controller 44 transitions to the execution state ST3 without checking or updating semaphore S and performs a convolution operation based on the instruction field IF.

[0173] After executing instruction 2, the state controller 44 transitions to the decode state ST2 and decodes instruction 3. In instruction 3, none of the semaphore operation fields SF are set to "1". Therefore, the state controller 44 transitions to the execution state ST3 without checking or updating semaphore S and performs the convolution operation based on the instruction field IF.

[0174] After executing instruction 3, the state controller 44 transitions to the decode state ST2 and decodes instruction 4. In instruction 4, none of the P operation fields are set to "1". Therefore, the state controller 44 transitions to the execution state ST3 without checking or updating the semaphore S and performs a convolution operation based on the instruction field IF.

[0175] If the V operation field is set to "1", the state controller 44 performs a V operation on the semaphore S corresponding to the V operation field set to "1" after the completion of the convolution operation of instruction 4. In instruction 4, the V operation field for the first write semaphore S1W and the V operation field for the second read semaphore S2R are set to "1". Therefore, after the completion of the convolution operation of instruction 4, the state controller 44 performs V operations on the first write semaphore S1W and the second read semaphore S2R.

[0176] After executing instruction 4, the state controller 44 transitions to idle state ST1, ending the execution of the series of convolution operation instructions consisting of 4 instructions.

[0177] When the convolutional operation circuit 4 uses the quantization output data written by the quantization operation circuit 5 to the first memory 1 as input data, the operation field corresponding to the third semaphore S3 is used.

[0178] The convolution operation instruction commands a convolution operation based on the instruction field IF, and also checks and updates the associated semaphore S based on the semaphore operation field SF. Since the instruction field IF and the semaphore operation field SF are contained within a single convolution operation instruction, the number of instruction commands C4 required to perform the convolution operation can be reduced. In addition, the processing time related to instruction execution, such as decoding, can be shortened.

[0179] [Quantization operation execution instruction] Figure 24 is a diagram illustrating the instruction to perform a quantization operation. The quantization operation execution instruction is one of the instruction commands C5 for the quantization operation circuit 5. The quantization operation execution instruction has an instruction field IF in which instructions for the quantization operation circuit 5 are stored, and a semaphore operation field SF in which operations on the semaphore S are stored. The instruction field IF and the semaphore operation field SF are contained in a single instruction as the quantization operation execution instruction.

[0180] The instruction field IF of the quantization operation instruction is the field that stores instructions for the quantization operation circuit 5. For example, the instruction field IF stores commands that cause the vector operation circuit 52 and the quantization circuit 53 to perform operations, as well as the size and memory address of the output data f and the quantization parameter p.

[0181] The semaphore operation field SF of the quantization operation instruction stores operations on the semaphore S associated with the instruction stored in the instruction field IF. The quantization operation circuit 5 is a Consumer in the second data flow F2 and a Producer in the third data flow F3. Therefore, the associated semaphores S are the second semaphore S2 and the third semaphore S3. As shown in Figure 24, the semaphore operation field SF of the quantization operation instruction includes operation fields for the second semaphore S2 and the third semaphore S3.

[0182] The second semaphore S2 for the second data flow F2, which the quantization arithmetic circuit 5 acts as the consumer, is provided with a P operation field for the read semaphore (S2R) and a V operation field for the write semaphore (S2W).

[0183] The third semaphore S3 for the third data flow F3, which is produced by the quantization arithmetic circuit 5, is provided with a P operation field for the write semaphore (S3W) and a V operation field for the read semaphore (S3R).

[0184] The state controller 54 of the quantization operation circuit 5 checks and updates the semaphore S in response to a quantization operation execution instruction in which the P operation field or V operation field is set to "1", similar to the operation of the state controller 44 for a convolution operation execution instruction.

[0185] [DMA Transfer Order] Figure 25 is a diagram illustrating the DMA transfer execution command. The DMA transfer execution command is one of the command commands C3 for DMAC3. The DMA transfer execution command has an instruction field IF that stores the instruction for DMAC3 and a semaphore operation field SF that stores operations on semaphore S. The instruction field IF and the semaphore operation field SF are contained in a single command as the DMA transfer execution command.

[0186] The instruction field IF of a DMA transfer execution instruction is the field where the instruction for DMAC3 is stored. For example, the instruction field IF stores the memory addresses of the memory source and destination, and the size of the data to be transferred.

[0187] The semaphore operation field SF of a DMA transfer execution command stores operations on the semaphore S associated with the command stored in the command field IF. DMAC3 is the Producer in the first data flow F1. Therefore, the associated semaphore S is the first semaphore S1. As shown in Figure 25, the semaphore operation field SF of the DMA transfer execution command includes an operation field for the first semaphore S1.

[0188] The first semaphore S1 for the first data flow F1, where DMAC3 acts as the producer, is provided with a P operation field for the write semaphore (S1W) and a V operation field for the read semaphore (S1R).

[0189] The DMAC3 state controller 32 checks and updates the semaphore S in response to a DMA transfer execution command in which the P operation field or V operation field is set to "1", similar to the operation of the state controller 44 for a convolution operation execution command.

[0190] According to the NN circuit 100 and the control method for the neural network circuit of this embodiment, the NN circuit 100, which can be embedded in embedded devices such as IoT devices, can be operated with high performance. In the convolution operation execution instruction, quantization operation execution instruction, and DMA transfer execution instruction, the instruction field IF and the semaphore operation field SF are contained in a single instruction. Therefore, the number of instruction commands for performing convolution operations, etc., can be reduced. In addition, the processing time related to instruction execution such as decoding can be shortened.

[0191] Although the first embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope of the gist of the present invention. Furthermore, the components shown in the above-described embodiment and modified examples can be combined as appropriate.

[0192] (Second embodiment) A neural network circuit 100B (hereinafter referred to as "NN circuit 100B") according to the second embodiment of the present invention will be described with reference to Figures 26 to 27. In the following description, components that are common to those already described will be denoted by the same reference numerals, and redundant explanations will be omitted.

[0193] The NN circuit 100B differs from the NN circuit 100 of the first embodiment in its instructions (convolution operation instruction, quantization operation execution instruction, DMA transfer execution instruction) and the control of the clock enable CE3, CE4, and CE5. In the first embodiment, the clock enable CE3, CE4, and CE5 were controlled in conjunction with the state transitions of the state controller. In the second embodiment, the clock enable CE3, CE4, and CE5 are controlled based on the instructions (convolution operation instruction, quantization operation execution instruction, DMA transfer execution instruction). Hereafter, the control of the convolution operation instruction and the clock enable CE4 will be explained as an example.

[0194] Figure 26 shows a convolution operation instruction. In this embodiment, the convolution operation instruction further includes an instruction field IF and a semaphore operation field SF, as well as a clock enable operation field CF which stores operations on the clock enable CE.

[0195] The clock enable operation field CF stores operations such as controlling the clock enable CE4 when the semaphore S corresponding to the P operation field set to "1" does not meet the execution conditions and a wait occurs in the convolution arithmetic circuit 4. Specifically, when the clock enable operation field CF is "1" and a wait occurs, the control of the clock enable CE4 is activated and clock gating is performed. When the clock enable operation field CF is "0" and a wait occurs, the control of the clock enable CE4 is deactivated and clock gating is not performed.

[0196] Figure 27 shows a specific example of a convolution operation instruction. The state controller 44 of the convolution arithmetic circuit 4 transitions to the decode state ST2 and decodes the convolution arithmetic instruction A shown in Figure 27, which is stored in the instruction queue 45.

[0197] If the P operation field is set to "1", the state controller 44 reads the semaphore S corresponding to the P operation field set to "1" from the controller 6 via the internal bus IB and determines whether the execution condition is met. The execution condition is that all semaphores S corresponding to the P operation field set to "1" are "1" or greater. In the convolution operation instruction A, the P operation field for the first read semaphore S1R is set to "1". Therefore, the state controller 44 reads the first read semaphore S1R and determines whether the execution condition is met.

[0198] If the P operation field is set to "1", the state controller 44 waits until the semaphore S corresponding to the P operation field set to "1" is updated and the execution conditions are met. In the case of a convolution operation instruction A, the state controller 44 waits until the semaphore S is updated and the execution conditions are met (Wait) unless the first read semaphore S1R is "1" or greater (Not Ready).

[0199] During the period when the convolution arithmetic circuit 4 is waiting, if the clock enable operation field CF is set to "1 (clock gating enabled)", the clock control unit 49 negates the clock enable signal CE4 and sets it to disabled. As a result, the toggling of the gated clock GC4 is stopped, and the dynamic power associated with the gated clock GC4 can be reduced.

[0200] During the period when the convolution arithmetic circuit 4 is waiting, if the clock enable operation field CF is set to "0 (clock gating disabled)", the clock control unit 49 maintains the enabled setting instead of negating the clock enable signal CE4 and setting it to disabled.

[0201] When semaphore S is updated and the conditions for implementation are met, the clock control unit 49 asserts the clock enable signal CE4 to enable it. As a result, the toggling of the clock-gated clock GC4 is resumed.

[0202] Similarly, the quantization operation execution instruction may further have a clock enable operation field CF in which operations such as controlling the clock enable CE5 are stored. Similarly, the DMA transfer execution instruction may further have a clock enable operation field CF in which operations such as controlling the clock enable CE3 are stored.

[0203] According to the NN circuit 100B and the control method for the neural network circuit of this embodiment, the NN circuit 100, which can be embedded in embedded devices such as IoT devices, can be operated with high performance. In the convolution operation execution instruction, quantization operation execution instruction, and DMA transfer execution instruction, the instruction field IF, the semaphore operation field SF, and the clock enable operation field CF are contained in a single instruction. Therefore, the number of instruction commands for performing convolution operations, etc., can be reduced. In addition, the processing time related to instruction execution such as decoding can be shortened.

[0204] According to the NN circuit 100B and the control method for the neural network circuit of this embodiment, the enabling and disabling of clock gating can be set for each instruction. For example, fine-grained control of clock gating can be implemented, such as disabling clock gating only for some instructions when a predetermined combination of instructions is used.

[0205] For example, even if semaphore S does not meet the execution conditions and a wait occurs in one of the circuits, if it can be determined in advance that the waiting time will be short, the clock gating setting may be kept enabled instead of being disabled. When clock gating is disabled (clock toggling is stopped) and then re-enabled (clock toggling is restarted), a predetermined waiting time is required for the circuit to resume operation. As mentioned above, if it can be determined in advance that the waiting time will be short, the overall processing time may be shortened by keeping the clock gating enabled setting and eliminating the predetermined waiting time for operation to resume due to clock gating activation.

[0206] Although a second embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are also included within the scope of the gist of the present invention. Furthermore, the components shown in the above-described embodiments and modifications can be combined as appropriate.

[0207] (Variation 1) In the above embodiment, an example of an instruction in which multiple semaphore operation fields SF are contained within a single instruction field IF is shown, but the instruction configuration is not limited to this. An instruction may also contain multiple instruction fields IF and multiple semaphore operation fields SF associated with each instruction field IF within a single instruction. Furthermore, the method of containing the instruction field IF and semaphore operation fields SF in a single instruction is not limited to the configuration of the above embodiment. Moreover, the instruction field IF and semaphore operation fields SF may be divided and contained within multiple instructions. The same effect can be achieved as long as the instruction field IF is associated with the corresponding semaphore operation field SF in the instruction.

[0208] (Modification 2) In the above embodiment, the first memory 1 and the second memory 2 were separate memories, but the configurations of the first memory 1 and the second memory 2 are not limited to this. For example, the first memory 1 and the second memory 2 may be the first memory area and the second memory area of ​​the same memory.

[0209] (Variation 3) In the above embodiment, the semaphore S was provided for the first data flow F1, the second data flow F2, and the third data flow F3, but the form of the semaphore S is not limited thereto. For example, the semaphore S may be provided for the data flow in which the DMAC3 writes weight w to the weight memory 41 and the multiplier 42 reads out weight w. For example, the semaphore S may be provided for the data flow in which the DMAC3 writes quantization parameter q to the quantization parameter memory 51 and the quantization circuit 53 reads out quantization parameter q.

[0210] (Modification 4) For example, the data input to the NN circuit 100 described in the above embodiment is not limited to a single format, but can consist of still images, moving images, audio, text, numerical values, and combinations thereof. Furthermore, the data input to the NN circuit 100 is not limited to measurement results from physical quantity measuring instruments such as optical sensors, thermometers, Global Positioning System (GPS) measuring instruments, angular velocity measuring instruments, and anemometers that can be mounted on the edge device on which the NN circuit 100 is provided. It may also be a combination of peripheral information such as base station information, vehicle / ship information, weather information, congestion information, financial information, and personal information received from peripheral devices via wired or wireless communication.

[0211] (Variation 5) Edge devices equipped with the NN circuit 100 are envisioned to include mobile devices such as mobile phones powered by batteries, smart devices such as personal computers, digital cameras, game consoles, and robotic products, but are not limited to these. It can also be used in products with a high demand for peak power limiting via Power on Ethernet (PoE), reduced product heat generation, or extended operating time, yielding effects not seen in other prior examples. For example, applying it to in-vehicle cameras mounted on vehicles and ships, or surveillance cameras installed in public facilities and on roads, not only enables long-duration shooting but also contributes to weight reduction and increased durability. Similar effects can also be achieved by applying it to display devices such as televisions and displays, medical equipment such as medical cameras and surgical robots, and work robots used in manufacturing and construction sites.

[0212] (Experimental variation 6) The NN circuit 100 may be implemented using one or more processors, either partially or entirely. For example, the NN circuit 100 may have part or all of its input or output layer implemented by software processing by a processor. The part of the input or output layer implemented by software processing could be, for example, data normalization or transformation. This allows for support of various input or output formats. The software executed by the processor may be configured to be rewritable using communication means or external media.

[0213] (Example 7) The NN circuit 100 may implement part of the processing in CNN200 by combining it with a Graphics Processing Unit (GPU) or the like on the cloud. The NN circuit 100 can perform more complex processing with fewer resources by performing processing on the cloud in addition to processing performed on the edge device where the NN circuit 100 is installed, or by performing processing on the edge device in addition to processing on the cloud. With such a configuration, the NN circuit 100 can reduce the amount of communication between the edge device and the cloud through distributed processing.

[0214] (Variation 8) The operations performed by the NN circuit 100 were at least a part of the trained CNN 200, but the scope of operations performed by the NN circuit 100 is not limited to this. The operations performed by the NN circuit 100 may also be at least a part of a trained neural network that repeats two types of operations, such as convolution and quantization.

[0215] (Extreme variation 9) In the first and second embodiments described above, examples were shown in which the control of clock enable is controlled based on state transitions or instructions of the state controller. However, the control target based on state transitions or instructions of the state controller is not limited to clock enable. For example, the power supply to each component included in the NN circuit 100 shown in Figure 4 may be the control target. Specifically, a power control unit is provided to control the power supplied to each component included in the NN circuit 100 (convolution arithmetic circuit 4 and quantization arithmetic circuit 5). The power control unit controls the enable signal for power supply (power gating control) based on state transitions or instructions of the state controller. For example, the power control unit controls the power supply to each component in accordance with state transitions such as Idle and Wait in the state controller. Note that this power supply control may be combined with the clock enable control of the embodiments described above. When power supply is stopped and then restarted, the circuit requires a predetermined amount of time to recover. Therefore, the clock enable (stop the clock toggling) and the enable signal for power supply (stop the power supply) may be disabled only when it can be determined in advance that the waiting time required when the semaphore S does not meet the implementation conditions is longer than a predetermined period.

[0216] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that are obvious to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein. [Industrial applicability]

[0217] This invention can be applied to neural network computations. [Explanation of Symbols]

[0218] 200 Convolutional Neural Networks 100 Neural Network Circuits (NN Circuits) 1 First Memory 2 Second Memory 3. DMA Controller (DMAC) 4. Convolutional arithmetic circuit 42 Multipliers 43. Accumulator Circuit 5 Quantization operation circuit 52 Vector Operation Circuits 53 Quantization circuit 6 Controllers 61 registers S semaphore F1 First Dataflow F2 Second Dataflow F3 Third Dataflow GC3 Gated Clock (Third Clock) GC4 Gated Clock (First Clock) GC5 Gated Clock (Second Clock)

Claims

1. A convolution circuit that performs a convolution operation on input data, A quantization circuit that performs a quantization operation on the convolution operation output data of the aforementioned convolution operation circuit, Equipped with, When the convolution circuit waits to perform the convolution operation, it enables clock gating of the first clock supplied to at least a portion of the convolution circuit. Neural network circuit.

2. When the quantization circuit waits to execute the quantization operation, it enables clock gating of the second clock supplied to at least a portion of the quantization circuit. The neural network circuit according to claim 1.

3. A first memory for storing the aforementioned input data, A second memory for storing the output data of the aforementioned convolution operation, It further possesses, When the convolution operation circuit performs the convolution operation on the input data stored in the first memory, it disables the clock gating of the first clock. When the quantization circuit performs the quantization operation on the convolution operation output data stored in the second memory, it disables the clock gating of the second clock. The neural network circuit according to claim 2.

4. The quantization output data of the quantization circuit is stored in the first memory. The quantization operation output data stored in the first memory is input to the convolution operation circuit as input data. The neural network circuit according to claim 3.

5. A third read semaphore that restricts reading from the first memory by the convolution arithmetic circuit, A second write semaphore that restricts writing to the second memory by the convolution arithmetic circuit, Equipped with, When the convolution circuit waits to execute the convolution operation based on the third read semaphore and the second write semaphore, it enables clock gating of the first clock supplied to at least a portion of the convolution circuit. The neural network circuit according to claim 3 or claim 4.

6. A second read semaphore that restricts reading from the second memory by the quantization arithmetic circuit, A third write semaphore that restricts writing to the first memory by the quantization arithmetic circuit, Furthermore, When the quantization circuit waits to execute the quantization operation based on the second read semaphore and the third write semaphore, it enables clock gating of the second clock supplied to at least a portion of the quantization circuit. The neural network circuit according to claim 5.

7. A DMA controller that transfers the input data to the first memory, A first read semaphore that restricts reading from the first memory by the convolution arithmetic circuit, Furthermore, When the convolution circuit waits to execute the convolution operation based on the first read semaphore and the second write semaphore, it enables clock gating of the first clock supplied to at least a portion of the convolution circuit. The neural network circuit according to claim 5 or claim 6.

8. A first write semaphore that restricts writing to the first memory by the DMA controller, Furthermore, When the DMA controller waits for the execution of the transfer based on the first write semaphore, it enables clock gating of the third clock supplied to at least a portion of the DMA controller. The neural network circuit according to claim 7.

9. The circuit includes a control circuit that executes a convolution operation execution instruction that commands the convolution operation circuit to perform, in a single instruction, a determination of the conditions for performing the convolution operation based on the third read semaphore and the second write semaphore, the performance of the convolution operation based on the determination, and whether or not to perform the clock gating on the convolution operation circuit based on the determination. The neural network circuit according to claim 5.

10. The control circuit executes a quantization operation execution instruction that commands the quantization operation circuit to perform, in a single instruction, a determination of the conditions for performing the quantization operation based on the second read semaphore and the third write semaphore, the performance of the quantization operation based on the determination, and whether or not to perform the clock gating on the quantization operation circuit based on the determination. The neural network circuit according to claim 6.

11. The convolution operation circuit is provided with a control circuit that executes a convolution operation execution instruction that commands, in a single instruction, to determine the conditions for performing the convolution operation based on the first read semaphore and the second write semaphore, to perform the convolution operation based on the determination, and to determine whether or not to perform the clock gating based on the determination. The neural network circuit according to claim 7.

12. The DMA controller is provided with a control circuit that executes a DMA transfer execution command that commands, in a single instruction, to determine the conditions for performing the convolution operation based on the first write semaphore, to perform the transfer based on the determination, and to determine whether or not to perform the clock gating based on the determination. The neural network circuit according to claim 8.

13. A convolution circuit that performs a convolution operation on input data, A quantization circuit that performs a quantization operation on the convolution operation output data of the aforementioned convolution operation circuit, A control method for a neural network circuit comprising, When the convolution circuit waits to execute the convolution operation, the clock gating of the first clock supplied to at least a portion of the convolution circuit is enabled. A method for controlling a neural network circuit.

14. When the quantization circuit waits to execute the quantization operation, the clock gating of the second clock supplied to at least a portion of the quantization circuit is enabled. A method for controlling a neural network circuit according to claim 13.

15. The aforementioned neural network circuit is A first memory for storing the aforementioned input data, A second memory for storing the output data of the aforementioned convolution operation, It further possesses, When the convolution operation circuit performs the convolution operation on the input data stored in the first memory, the clock gating of the first clock is disabled. When the quantization circuit performs the quantization operation on the convolution operation output data stored in the second memory, the clock gating of the second clock is disabled. A method for controlling a neural network circuit according to claim 14.

Images