Neural network circuit and neural network chip

Abstract

The present disclosure relates to a neural network circuit and a neural network chip. The neural network circuit comprises M execution modules and a control module. Each of the M execution modules stores different input channel corresponding feature maps and convolution kernels, and the feature maps and the convolution kernels in the M execution modules are stored in the same physical address. The control module is configured to receive a first instruction and transmit a control signal to the M execution modules according to the first instruction. Each of the M execution modules is configured to read the stored first feature maps and first convolution kernels from a target physical address in response to the received control signal, and perform convolution operation on the first feature maps based on the first convolution kernels. The present disclosure can improve the calculation efficiency and has a more extensive application scenario.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of semiconductor, and in particular, to a neural network circuit and a neural network chip. BACKGROUND

[0002] With the continuous development of technology, great achievements have been made in deep learning and artificial intelligence, and there is an increasingly high demand for the processing speed of various deep learning and artificial intelligence algorithms. Therefore, application specific integrated circuit (ASIC) chips dedicated to neural network acceleration have emerged.

[0003] The existing tensor processing unit (TPU) is a commonly used neural network acceleration ASIC chip. The chip has an architecture dedicated to deep learning and can accelerate the efficiency of neural network related calculations. However, there is still room for optimization. SUMMARY

[0004] The present disclosure provides a neural network circuit and a neural network chip to realize hardware acceleration of convolution operation and support multiple data processing types.

[0005] In a first aspect, the present disclosure provides a neural network circuit, comprising: M execution modules and a control module; wherein each of the M execution modules stores feature maps corresponding to different input channels and convolution kernels, and the physical addresses of the feature maps and the convolution kernels stored in the M execution modules are the same, and M is an integer greater than or equal to 2. And wherein the control module is configured to: receive a first instruction sent by an external controller, the first instruction being generated by the external controller; and transmit a control signal to the M execution modules according to the first instruction, wherein the control signal is used to indicate a target physical address in the M execution modules. And each of the M execution modules is configured to: in response to the received control signal, read the stored first feature map and first convolution kernel from the target physical address; and perform convolution operation on the first feature map based on the first convolution kernel.

[0006] In a possible implementation, the convolution operation includes at least one of the following: pooling convolution and depth convolution.

[0007] In a possible implementation, each of the M execution modules comprises a data control submodule, a data storage submodule, and a data calculation submodule, wherein the first feature map and the first convolution kernel are stored in the data storage submodule, and wherein the data control submodule is configured to: receive a control signal transmitted by the control module; obtain a target physical address according to the control signal; read the first feature map and the first convolution kernel from the data storage submodule according to the target physical address; and input the read first feature map and first convolution kernel into the data calculation submodule, and the data calculation submodule is configured to: receive the first feature map and the first convolution kernel input by the data control submodule; and perform convolution operation on the first feature map based on the first convolution kernel.

[0008] In a possible implementation, the data calculation submodule is further configured to output the first feature map after convolution to the data control submodule, and the data control submodule is further configured to write the first feature map after convolution into the data storage submodule.

[0009] In a possible implementation, each of the M execution modules is further configured to perform a target operation on the first feature map and / or the first convolution kernel, and perform the convolution operation based on the first convolution kernel and / or the first feature map after operation.

[0010] In a possible implementation, each of the M execution modules is further configured to perform at least one of: performing the convolution operation on the first feature map based on the first convolution kernel after operation; performing the convolution operation on the first feature map after operation based on the first convolution kernel; and performing the convolution operation on the first feature map after operation based on the first convolution kernel after operation.

[0011] In a possible implementation, the target operation comprises at least one of: an interpolation operation on the feature map, a padding operation on the feature map, a dilation operation on the convolution kernel, and a rotation operation on the convolution kernel.

[0012] In a possible implementation, the control module is further configured to input an operand required by the target operation to each of the M execution modules.

[0013] In a possible implementation, the control module is further configured to instruct each of the M execution modules to obtain an operand required by the target operation in the data storage submodule.

[0014] In a second aspect, the present disclosure provides a neural network chip, comprising: a substrate; and a neural network circuit as in the first aspect and possible implementation modes thereof disposed on the substrate.

[0015] The technical solutions provided by the present disclosure can have the following beneficial effects:

[0016] In the present disclosure, the M execution modules are configured to store feature maps and convolution kernels of different channels, and the physical addresses of the feature maps and the convolution kernels stored in the M execution modules are the same, so that the M execution modules simultaneously perform convolution operations on corresponding data under the control of the control module. On the one hand, the parallel processing of multi-channel pooling convolution and deep convolution can be realized based on the optimized architecture of the neural network circuit, and the calculation efficiency is improved. On the other hand, various requirements in the process of pooling convolution and deep convolution can be effectively supported, so that the neural network circuit has a more extensive application scenario.

[0017] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the protection scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure.

[0019] Figure 1 It is a structural schematic diagram of a systolic array in the related art.

[0020] Figure 2 It is an architectural schematic diagram of a neural network circuit in an embodiment of the present disclosure.

[0021] Figure 3 It is an architectural schematic diagram of an execution module in an embodiment of the present disclosure.

[0022] Figure 4 It is a schematic diagram of a feature map and a convolution kernel in an embodiment of the present disclosure.

[0023] Figure 5 It is an architectural schematic diagram of a data storage submodule in an embodiment of the present disclosure.

[0024] Figure 6 It is a schematic diagram of input data in an embodiment of the present disclosure.

[0025] Figure 7 It is a schematic diagram of the process of convolution processing in an embodiment of the present disclosure.

[0026] Figure 8 It is a schematic diagram of the process of interpolation and padding in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses consistent with some aspects of this disclosure as detailed in the appended claims.

[0028] To illustrate the technical solutions described in this disclosure, specific embodiments are provided below.

[0029] With the continuous development of science and technology, great achievements have been made in deep learning and artificial intelligence, which has led to an increasing demand for the processing speed of various deep learning and artificial intelligence algorithms. As a result, application-specific integrated circuit (ASIC) chips have emerged, which are specifically designed for accelerating neural networks.

[0030] Tensor processing units (TPUs) are commonly used ASIC chips for accelerating neural networks. TPUs have a dedicated architecture for deep learning, with their core computing units employing a 256x256 systolic array. Compared to traditional architectures, systolic arrays offer two advantages: first, data sharing between processing elements (PEs) shortens the critical path and thus reduces clock cycle time; second, it reduces the number of reads from static random-access memory (SRAM), saving energy. Figure 1 This is a schematic diagram of a pulsating array structure in related technologies, such as... Figure 1 As shown, a box represents a computational unit, and multiple computational units are arranged in an array. The most frequent operation in neural network computation is convolution, which is essentially matrix multiplication. When using a systolic array to solve this matrix multiplication problem, the weights (denoted by matrix coordinates W11 to W44) can be pre-loaded. Input data (denoted by matrix coordinates a11 to a44) enters from the left, and output data is output from the bottom. The input data is staggered in different column directions; for example, the columns containing a12 to a42 enter the computational unit one clock cycle later than the columns containing a11 to a41. This causes the input data and weights to change in different clock cycles within each computational unit, thus forming a continuous pipeline. This allows matrix multiplication to be output in a shorter, rhythmic manner, making it highly efficient.

[0031] It should be noted that, Figure 1The "→" shown represents the data transmission direction of the input data, and the "↓" represents the data transmission direction after the input data is calculated with the weight.

[0032] Pooling is an important part of convolutional neural network calculation, and its main purpose is to compress the input feature map. On the one hand, the feature map is made smaller to simplify the network calculation complexity, and on the other hand, the main features are extracted through feature compression.

[0033] However, since the input data of the above-mentioned TPU enters different calculation units in a pipelined manner, there is a problem of fixed calculation mode. On the one hand, when facing convolution kernel layers that may have more convolution calculations such as pooling convolution and depthwise convolution, multiple clock cycles are needed to calculate different layer data, and the calculation efficiency is reduced. On the other hand, for data processing types such as feature map interpolation, feature map padding, convolution kernel dilation, convolution kernel rotation, and region of interest (ROI) pooling and depthwise convolution in neural network operation, the applicability is poor compared to the traditional architecture.

[0034] In order to solve the above-mentioned problems, the neural network circuit provided in the embodiments of the present disclosure is used to realize the hardware acceleration of convolution operation and support multiple data processing types.

[0035] Figure 2 The architecture of a neural network circuit in the embodiments of the present disclosure is shown in the figure. As shown in the figure, the neural network circuit can include M execution modules (denoted as execution module Lane0 to execution module Lanen) and a control module 201. Wherein, the M execution modules and the control module 201 are connected by a bus to form a data path. Each of the M execution modules is used for data storage and calculation, and the control module is used for instruction analysis and logical control of the calculation process. Figure 2

[0036] The above-mentioned control module 201 is used for receiving a first instruction sent by an external controller, and transmitting a control signal to the M execution modules according to the first instruction. The first instruction is used to indicate the neural network calculation, and the control signal is used to indicate the target physical address in the M execution modules.

[0037] It can be understood that when the neural network circuit performs neural network calculation, a single instruction multiple data processing process can be used. First, the control module receives the first instruction indicating the convolution calculation generated by the external controller through the bus, second, the control module generates the control signal according to the first instruction, and finally, the control module can transmit the control signal to the M execution modules through the bus.

[0038] ​The control signal is used to indicate a target physical address, and the target physical address is a physical storage address of data requiring convolution calculation in the M execution modules. Here, the data requiring convolution calculation includes neural network data to be calculated, such as image data to be recognized, sound data, etc., and the data requiring convolution calculation also includes parameters for neural network calculation, such as offline trained neural network parameters, etc. It can be seen that the data requiring convolution calculation includes feature maps and convolution kernels.

[0039] Each of the M execution modules is configured to read a first feature map and a first convolution kernel stored by itself from the target physical address in response to the received control signal, and perform convolution operation on the first feature map based on the first convolution kernel.

[0040] It can be understood that the control signal includes the target physical address carrying the feature map and the convolution kernel. When the i th execution module in the M execution modules receives the control signal, the i th execution module can read the feature map and the convolution kernel stored by itself at the target physical address in response to the control signal, and perform convolution operation on the first feature map based on the first convolution kernel.

[0041] Here, different execution modules in the M execution modules store feature maps and convolution kernels corresponding to different input channels, that is, the i th execution module in the M execution modules stores feature maps and convolution kernels corresponding to the i th input channel. The target physical addresses of the feature maps and the convolution kernels stored in the M execution modules are the same, and the value of i is an integer from 1 to M, and the value of M is greater than or equal to 2.

[0042] It should be noted that the convolution operation includes at least one of the following: pooling convolution, depth convolution.

[0043] It can be understood that in the calculation of the neural network, the convolution calculation corresponding to the pooling convolution and the depth convolution is the multiplication and addition operation of a channel of feature maps and a channel of convolution kernels. Therefore, in order to adapt the neural network circuit of the present disclosure to the pooling convolution and the depth convolution and improve the calculation efficiency, each of the M execution modules needs to store the corresponding feature map and convolution kernel of one input channel to realize the data parallel calculation of multiple channels. Further, the physical addresses of the feature maps and the convolution kernels stored in each of the M execution modules can be the same, and when the control signal gives the target physical address, the M execution modules can read the feature maps and the convolution kernels according to the same target physical address, thereby simplifying the instructions of the neural network circuit and improving the efficiency.

[0044] In the embodiments of the present disclosure, the M execution modules are configured to store feature maps and convolution kernels of different channels, and the target physical addresses of the feature maps and the convolution kernels stored in the M execution modules are the same, so that the M execution modules simultaneously perform pooling processing on corresponding data through the control of the control module. On the one hand, the parallel processing of multi-channel pooling can be realized based on the optimized architecture of the neural network circuit, and the calculation efficiency is improved. On the other hand, various requirements in the pooling process can be effectively supported, so that the neural network circuit has a more extensive application scenario.

[0045] In some possible implementation manners, the execution module can adopt the architecture shown in FIG. 3 to perform convolution operation on the feature map and the convolution kernel in response to the control signal. Figure 3 In some possible implementation manners, the execution module can adopt the architecture shown in FIG. 3 to perform convolution operation on the feature map and the convolution kernel in response to the control signal. Figure 3 FIG. 3 shows an architecture of an execution module in the embodiments of the present disclosure. As shown in FIG. 3, each of the M execution modules 300 includes a data control submodule 301, a data storage submodule 302, and a data calculation submodule 303. The data control submodule 301 is connected with the data storage submodule 302 and the data calculation submodule 303 to form a data path. Figure 3

[0046] The data control submodule 301 is configured to receive a control signal transmitted by the control module, and obtain a target physical address according to the control signal. The data control submodule 301 is further configured to read a first feature map and a first convolution kernel from the data storage submodule 302 according to the target physical address, and input the read first feature map and the first convolution kernel into the data calculation submodule 303.

[0047] It can be understood that the data control submodule 301 in the execution module can receive the control signal input by the control module through the data path between the execution module and the control module. The data control submodule 301 can obtain the target physical address by analyzing the control signal. After the data control submodule 301 obtains the target physical address, the data control submodule 301 can read the feature map and the convolution kernel stored in the data storage submodule 302 based on the target physical address. The data control submodule 301 can further input the read feature map and convolution kernel into the data calculation submodule 303, and receive the feature map obtained by the data calculation submodule 303 after the feature map and the convolution kernel are pooled. Finally, the data control submodule 301 inputs the first feature map after convolution into the data storage submodule 302, so as to complete a convolution operation.

[0048] It should be noted that each of the M execution modules processes data by using the same method as described above when receiving the control signal input by the control module, so that the M execution modules perform parallel processing on convolution operations of different channels based on one control signal, and the calculation efficiency is accelerated.

[0049] ​The data storage submodule 302 is configured to store the feature map and the convolution kernel, and is further configured to store the first feature map after convolution input by the data control submodule 301.

[0050] It can be understood that the data storage submodule 302 is configured to store the feature map, the convolution kernel and the feature map after convolution (i.e. the calculation result after convolution). The feature map and the convolution kernel can be configured in the data storage submodule 302 according to the above-mentioned manner of storing the corresponding feature map and convolution kernel of the corresponding input channel (which can be the i-th input channel) in each execution module (which can be the i-th execution module) through a bus. The configuration manner can adopt various manners, which are not limited in the embodiments of the present disclosure. Further, the data storage submodule 302 can be a static random-access memory (SRAM) or other types of storage circuits, so that the data storage submodule 302 can complete the storage and retrieval of data based on the indication of the data control submodule 301.

[0051] In some possible embodiments, the data storage submodule 302 comprises N storage units, and N is greater than or equal to 2; the N storage units are arranged in an array and have N physical addresses.

[0052] It can be understood that the data storage submodule 302 can include a plurality of storage units arranged in an array. Each of the plurality of storage units has a specific row / column address, that is, the number of storage units is equal to the number of physical addresses indicating the storage positions. When the feature maps and the convolution kernels of different channels are stored in different units, the physical addresses of the feature maps and the convolution kernels of each channel should be the same.

[0053] Further, the convolution kernel can have K elements, and the K elements can be respectively stored in K storage units, and K is a positive integer greater than 2. The feature map can have M elements, and the M elements can be respectively stored in M storage units, and M is a positive integer greater than 2.

[0054] In an embodiment, when the data control submodule 301 obtains the convolution kernel with K elements or the feature map with M elements stored in the data storage submodule 302, the data control submodule 301 can also obtain the coordinate position of each element in the convolution kernel or the feature map, so that the target physical address is associated with the coordinates of each element. That is, the data control submodule 301 can quickly determine the coordinates of any one element based on the target physical address. Since the data control submodule 301 can determine the coordinates of any one element based on the target physical address, the data control submodule 301 can perform data processing on the input matrix of the feature map or the convolution kernel based on the target physical address, such as feature map padding and convolution kernel transposition.

[0055] For example, Figure 4 This is a schematic diagram of a feature map and a convolution kernel in an embodiment of this disclosure. Figure 5 This is a schematic diagram of the architecture of a data storage submodule according to an embodiment of this disclosure. Figure 4 As shown, a three-channel feature map 401 and a three-channel convolutional kernel 402 are convolved. Within each channel, each small square represents an element. Pooling a 4*5 feature map 401 and a 3*2 convolutional kernel 402 yields a 2*4 output feature map 403. During storage, the feature map 401 and the convolutional kernel 402 are each divided into 20 elements. Figure 4 The leftmost number shows 1 to 20) and 6 elements ( Figure 4 (1 to 6 are shown in the middle). Each element is stored in a storage unit.

[0056] For example Figure 5 As shown, the M execution modules can be execution modules Lane0 to Laneen, and each of the execution modules Lane0 to Laneen has a data storage submodule 302. The structure of each data storage submodule 302 consists of N storage units 501. Figure 5 The elements 1 to N are arranged in an array. Each of the above elements occupies one of the N storage units 501. When M execution modules 300 are configured to store feature maps and convolution kernels for different channels, the feature maps and convolution kernels for different channels (e.g., the first element in the feature map) have the same physical address in each data storage submodule 302, such as... Figure 4 The position of the shadow.

[0057] The aforementioned data calculation submodule 303 is used to receive the first feature map and the first convolution kernel input by the data control submodule 301; and to perform convolution operation on the first feature map based on the first convolution kernel.

[0058] Understandably, the data computation submodule 303 can receive the first feature map and the first convolution kernel input from the data control submodule 301, and perform convolution operations based on the feature map and the convolution kernel. The convolved first feature map can also be output to the data control submodule 301.

[0059] In some possible implementations, in order to further improve the speed of convolution operation on data in the same channel within the execution module, the data calculation submodule 303 may include: L calculation components, where the value of L is greater than or equal to 2.

[0060] It can be understood that a plurality of computing components are included in the data computing submodule 303, and the plurality of computing components are independent of each other. When different feature map elements and convolution kernel elements are input into the plurality of components, the plurality of components can be processed in parallel to speed up the calculation.

[0061] In some possible implementation, since the data computing submodule 303 includes a plurality of computing components, the data control submodule 301 is further configured to input the jth element in the K convolution kernel elements into the N computing components simultaneously, and input the N feature map elements corresponding to the jth element into the N computing components respectively, so that the N computing components complete the convolution operation of the jth element and the N feature map elements simultaneously.

[0062] It can be understood that in the pooling operation, the corresponding relationship between the input feature map and the convolution kernel is determined by the size of the feature map and the convolution kernel in the column direction, and the step value of the convolution kernel in the column direction. Therefore, when the control module learns the number of elements of the feature map and the convolution kernel and the parameters of the convolution kernel, the control module can obtain the feature map corresponding to the convolution kernel. After determining the number (N) of feature map elements corresponding to the jth element in the convolution kernel, the control module can control the data control submodule 301 to input the jth element in the convolution kernel into a plurality of (N) computing components corresponding to the number (N) of feature map elements, and then control the data control submodule 301 to input the N feature map elements into the N computing components respectively, so that the N computing components complete the convolution operation of the N feature map elements and the same convolution kernel element simultaneously.

[0063] For example, as shown in the above Figure 4 , the feature map elements corresponding to the convolution kernel element marked as 1 are 1, 2, 5, 6, 9, 10, 13, and 14. The data control submodule 301 can input the convolution kernel element marked as 1 into eight computing components, and input the feature map elements marked as 1, 2, 5, 6, 9, 10, 13, and 14 into the above eight computing components respectively, so that the convolution kernel element marked as 1 respectively performs multiplication operation with the feature map elements marked as 1, 2, 5, 6, 9, 10, 13, and 14, and completes the convolution operation.

[0064] Here, each of the N computing components includes a multiplication operation circuit, an addition operation circuit, and a comparison operation circuit. Through the combination of these logic circuits, the convolution operation can be realized.

[0065] In some possible implementation, the N computing components can complete the convolution operation of the feature map elements and the convolution kernel elements in parallel. When the feature map includes a plurality of elements or the convolution kernel includes a plurality of elements, the control signal is further used to instruct the data control submodule 301 to input different feature map elements and convolution kernel elements into the data computing submodule 303 in different clock cycles.

[0066] It can be understood that the control signal is used to indicate the execution time. In an embodiment, the execution time can be the time at which the data control submodule 301 sends the feature map elements and the convolution kernel elements to the data calculation submodule 303 according to the control signal received; in another embodiment, the execution time can be the time at which the data control submodule 301 requests to obtain the feature map elements and the convolution kernel elements from the data storage submodule 302 according to the control signal received. When the time parameter and the target physical address are included in the control signal, the control signal received by the data control submodule 301 indicates that different feature map elements and convolution kernel elements are input to the data calculation submodule 303 at different clock cycles, so that the plurality of convolution kernel elements complete the pooling with the corresponding feature map elements.

[0067] In some possible embodiments, the data control submodule 301 inputs different feature map elements and convolution kernel elements to the data calculation submodule 303 at different clock cycles, including: at the kth clock cycle, the data control submodule 301 inputs the jth element in the K convolution kernel elements and the N feature map elements corresponding to the jth element to the data calculation submodule 303; at the k+1th clock cycle, the data control submodule 301 inputs the j+1th element in the K convolution kernel elements and the N feature map elements corresponding to the j+1th element to the data calculation submodule 303, and the value of k is equal to j.

[0068] It can be understood that when the clock cycle is indicated in the control signal, the data control submodule 301 can complete the convolution operation according to the clock cycle indicated by the control signal. The data control submodule 301 can control the feature map elements and the convolution kernel elements input to the data calculation submodule 303 by inputting the jth element in the K convolution kernel elements and the N feature map elements corresponding to the jth element to the data calculation submodule 303 at the kth clock cycle, and inputting the j+1th element in the K convolution kernel elements and the N feature map elements corresponding to the j+1th element to the data calculation submodule 303 at the k+1th clock cycle, so as to complete the convolution according to the clock cycle of the control signal, and the way can parallelly calculate a plurality of data, greatly reducing the calculation period required by the convolution operation.

[0069] An exemplary Figure 6 An exemplary diagram of input data in an embodiment of the present disclosure. In combination with the above description of the data control submodule 301, the data control submodule 301 can be understood as follows. Figure 4 The data control submodule 301 can be understood as follows. Figure 6As shown, there are 8 computing components in the data computing submodule 303 for parallel computing. Among them, each computing component has two input data, OP A and OPB, OP A represents the feature map elements input by the 8 computing components respectively, and OPB represents the convolution kernel elements input by the 8 computing components respectively. T represents a clock cycle, and T = i represents the i th clock cycle. That is, in the second clock cycle, the 8 computing components perform parallel computing of the pooling of the second convolution kernel element and the corresponding feature map element, in the third clock cycle, the 8 computing components perform parallel computing of the pooling of the third convolution kernel element and the corresponding feature map element, and so on, until the sixth clock cycle, the sixth convolution kernel element and the corresponding feature map element are pooled, and then the pooling of the feature map and the convolution kernel in one channel is completed.

[0070] In the embodiments of the present disclosure, the M execution modules are configured to store feature maps and convolution kernels of different channels, and the physical addresses of the feature maps and the convolution kernels stored in the M execution modules are the same, so that the M execution modules simultaneously perform pooling processing on the corresponding data under the control of the control module. In addition, the data computing submodule 303 in the M execution modules is respectively configured with L computing components, so that the L computing components simultaneously perform pooling processing on the convolution kernel elements and all the corresponding feature map elements under the control of the control module. On the one hand, the parallel processing of multi-channel pooling can be realized based on the optimized architecture of the neural network circuit, and the computing efficiency is improved; on the other hand, various needs in the pooling process can be effectively supported, so that the neural network circuit has a more extensive application scenario. In addition, the operation period of the pooling can be effectively reduced, and the computing time is greatly saved.

[0071] Next, the process of the pooling convolution based on the neural network circuit of the present disclosure is described with specific examples.

[0072] Figure 7 is a schematic diagram of the process of a convolution processing in the embodiments of the present disclosure. As shown in the figure, the neural network circuit includes a control module 301, an execution module 302, and a data computing submodule 303. The control module 301 is configured to control the execution module 302 to perform the pooling of the feature map and the convolution kernel in one channel, and the data computing submodule 303 is configured to perform the pooling of the feature map and the convolution kernel in one channel under the control of the control module 301. Figure 7As shown, each of the execution modules Lane0, Lane1 and Lane2 includes a data storage submodule (denoted as LMEM) and a data calculation submodule (denoted as EU). The LMEM is configured to store the input feature map 701 and the convolution kernel 702, and the output feature map 703 corresponding to the input feature map 701 and the convolution kernel 702 in the channel channel0, channel1 and channel2, respectively. For example, the execution module Lane0 stores the input feature map 01, the convolution kernel 02 and the output feature map 03 in the channel channel0. Similarly, the execution module Lane1 stores the input feature map 11, the convolution kernel 12 and the output feature map 13 in the channel channel1, and the execution module Lane2 stores the input feature map 21, the convolution kernel 22 and the output feature map 23 in the channel channel2. The input feature map 01, the input feature map 11 and the input feature map 21 together constitute the input feature map 701. As can be seen, the physical addresses of the feature maps and the convolution kernels stored in different execution modules are the same. Therefore, based on the physical address indicated by the same control signal, the three execution modules can retrieve the feature maps and the convolution kernels in the three channels, so that the execution modules Lane0, Lane1 and Lane2 can process the pooling processing of the three channels in parallel.

[0073] Further, each of the data calculation submodules EU in the three execution modules has eight calculation components, each of which has two input data ports, namely input OP A and input OPB. The data in the input OP A is from the feature map element (in the input feature map 01) stored in the data storage submodule LMEM, and the data in the input OPB is from the convolution kernel element stored in the data storage submodule LMEM. As can be seen, when the input OP A and the input OPB perform convolution calculation, one convolution kernel element and its corresponding eight feature map elements perform multiply and accumulate (MAC) operation at the same time, and finally the output feature map obtained by the multiply and accumulate operation is stored back into the data storage submodule LMEM.

[0074] Next, the convolution calculation of the three channels will be described in combination with the following figures. Figure 6 and Figure 7The first period is to calculate the product of the first convolution kernel element of the convolution kernel required to generate the output feature map and all corresponding input feature map elements (i.e. 1, 2, 5, 6, 9, 10, 13, 14). The second period is to calculate the product of the second convolution kernel element of the convolution kernel required to generate the output feature map and all corresponding input feature map elements (i.e. 2, 3, 6, 7, 10, 11, 14, 15), and add the product of the last period. In this way, the calculation of all convolution kernel elements and corresponding input feature map elements is completed in six periods, and the pooling processing of the feature map and the convolution kernel is realized.

[0075] Further, based on the multiplication operation circuit, the addition operation circuit and the comparison operation circuit configured in the calculation component, while completing the product calculation, the input OP A and the input OPB of different calculation components are marked by the mark bit in the form of marking, so that the calculation component that needs to perform the multiplication and accumulation is marked, and the calculation component performs the multiplication and accumulation operation. The calculation component obtains the multiplication and accumulation sum in all calculation periods, and the result of the deep convolution is obtained, so that the neural network circuit in the embodiment of the present disclosure supports efficient operation of deep convolution. Further, the calculation component multiplies the obtained multiplication and accumulation sum by the reciprocal of the number of periods corresponding to the multiplication and accumulation sum, and the result of average pooling is obtained, so that the neural network circuit in the embodiment of the present disclosure supports efficient operation of average pooling. In addition, since the comparison operation circuit is also configured in the calculation component, only the maximum products calculated in the current period and the last period are input into the comparison operation circuit for comparison, the larger number is selected, and the comparison with the products calculated in the subsequent periods is continued until the comparison of all periods is completed, the maximum product is obtained, and the max pooling is completed, so that the neural network circuit in the embodiment of the present disclosure supports efficient operation of max pooling.

[0076] As known from the above, through the neural network circuit in the embodiment of the present disclosure, the parallelism of data calculation is greatly expanded, the calculation time is reduced, and various types of convolution operations are supported, so that the practicability is greatly enhanced.

[0077] In some possible implementation manners, the i th execution module is further configured to perform a target operation on the first feature map and / or the first convolution kernel, and perform the convolution operation based on the first convolution kernel and / or the first feature map after the operation.

[0078] The target operation includes at least one of the following operations: an interpolation operation on the feature map, a padding operation on the feature map, a dilation operation on the convolution kernel, and a rotation operation on the convolution kernel.

[0079] It can be understood that in practical applications, there is a case where interpolation operation is required to be performed on the feature map, or padding operation is required to be performed on the feature map, or dilation operation is required to be performed on the convolution kernel, or rotation operation is required to be performed on the convolution kernel. In order to process the above cases, the neural network circuit in the embodiment of the present disclosure is further configured to perform the above operations by the i th execution module.

[0080] Here, the i th execution module performing the above operations can include at least one of the following three operations: first, performing convolution operation on the first feature map based on the first convolution kernel after operation; second, performing convolution operation on the first feature map after operation based on the first convolution kernel; and third, performing convolution operation on the first feature map after operation based on the first convolution kernel after operation.

[0081] In some possible implementations, the control module is further configured to input an operation number required by the target operation to each execution module, or instruct each execution module to obtain the operation number required by the target operation from the data storage submodule, so that each execution module performs the target operation.

[0082] Here, the operation number refers to data used for performing the target operation.

[0083] In the following, the process of the target operation performed by the i th execution module based on the neural network circuit of the present disclosure is described with specific examples.

[0084] In some embodiments, the data control submodule can operate on the feature map, that is, the data control submodule inputs a specific value or a null value to the data calculation submodule to change the original feature map.

[0085] It can be understood that since the physical address of the feature map element stored in the data storage submodule corresponds to the coordinate of the feature map element one by one, when interpolation operation is required to be performed on the feature map, only the interpolation in the x-axis and y-axis coordinates of the feature map needs to be configured, and then it can be known that the interpolation is inserted in which two feature map elements in the x direction and y direction of the feature map, respectively. The inserted value can be a specific value (constant), a null value, or data stored in the data storage submodule.

[0086] Similarly, when padding operation is required to be performed on the feature map, the feature map is also padded by the coordinate. When the coordinate is a padding coordinate point, the control signal instructs the data control submodule to input a padding value to the data calculation submodule. The padding value can be a specific value (constant), a null value, or data stored in the data storage submodule.

[0087] Exemplarily, Figure 8 A schematic diagram of an interpolation and padding processing process in the embodiment of the present disclosure is shown in FIG. 2. As shown in FIG. 2, the data control submodule inputs a specific value or a null value to the data calculation submodule according to the control signal, so that the data calculation submodule performs interpolation or padding operation on the feature map. Figure 8As shown, the control module can perform interpolation and padding on the 4*5 input feature map 801. As can be seen, the input feature map is padded with 1 on the top, bottom, left and right (the padding value is denoted as P), and is interpolated with 1 horizontally (the interpolation value is denoted as I). Subsequently, the control module can obtain the processed feature map 802. When the control module controls the execution module to perform data calculation, the data storage submodule LMEM in the execution module does not need to store the processed feature map, but only needs to store the original input feature map. The control module can generate corresponding control signals according to the coordinates of the processed feature map, so that the execution module inputs the feature map elements or the padding values into the calculation components in the data calculation submodule EU in different periods, to implement the pooling operation on the processed feature map. Figure 8 As shown, the input value of the calculation component in the first calculation period of the execution module is shown, and it can be seen that the feature map and the convolution kernel in the data storage submodule LMEM do not occupy more space, and do not need to be pre-processed by interpolation and padding deformation. The control module only needs to determine whether the calculation component is an interpolated and padded data, and indicates the same in the control signal.

[0088] In some embodiments, the data control submodule can operate on the convolution kernel, that is, the data control submodule inputs specific values or null values into the data calculation submodule, to change the original convolution kernel.

[0089] As can be understood, the convolution kernel inflation refers to inserting 0 in the width and height of the convolution kernel. When the neural network circuit in the embodiments of the present disclosure implements convolution kernel inflation, the inserted 0 convolution kernel does not need to be stored in the data storage submodule, like the above-mentioned processed feature map that does not need to be stored in the data storage submodule. Even because the convolution kernel inserted with 0 (i.e., the convolution kernel element is 0) does not contribute to the pooling result, when the convolution kernel element is 0 is needed in calculation, the control module can instruct the data control submodule to skip the input feature map corresponding to the convolution kernel element by the control signal, without inputting the calculation component, thereby reducing the calculation amount.

[0090] In some embodiments, the data control submodule can operate on the convolution kernel, which supports convolution kernel rotation.

[0091] As can be understood, the control module can obtain the coordinates of each convolution kernel element in the convolution kernel, and when the convolution kernel needs to be rotated, the control module can calculate the transformed convolution kernel coordinates by coordinate transformation. During calculation, the control module can generate control signals according to the transformed coordinates (corresponding physical addresses), so that the input calculation component is the rotated convolution kernel.

[0092] In some embodiments, the neural network circuit in the embodiments of the present disclosure also supports result extraction.

[0093] It can be understood that, for the extraction of the calculation result, the completed result is selected and saved. For such a calculation method, in the implementation process of the neural network circuit in the embodiment of the disclosure, all feature map elements and convolution kernel elements do not need to be calculated first, only the sliding step of the convolution kernel window needs to be changed, the calculation process that is not saved after original calculation can be omitted, and the calculation efficiency can be greatly improved.

[0094] In some embodiments, the neural network circuit in the embodiment of the disclosure also supports ROI operation.

[0095] It can be understood that, in the implementation process of the ROI operation of the neural network circuit in the embodiment of the disclosure, the ROI window coordinates are stored in the data storage submodule. Only the coordinates of all ROI windows need to be collected, the physical address of the window in each data storage submodule is provided according to the coordinates, the feature map elements are read and then composed into a new input feature map, and then the convolution operation in the above embodiment is performed with the convolution kernel.

[0096] As can be seen from the above embodiments, the neural network circuit in the embodiment of the disclosure can input the operation number to the data control submodule in the execution module by the control module, or instruct the data control submodule in the execution module to call the operation number from the data storage submodule, so that the execution module can change the data of the input data calculation submodule. The execution module changes the data of the input data calculation submodule to implement the target operation on the first feature map and / or the first convolution kernel, and performs the convolution operation based on the first convolution kernel and / or the first feature map after the operation. In this process, a variety of types of convolution operations are supported, and the practicability is greatly enhanced.

[0097] Based on the same inventive concept, the embodiment of the disclosure also provides a neural network chip, which can include a substrate and at least one above Figures 2 to 8 The neural network circuit in the corresponding embodiment, wherein the neural network circuit is arranged on the substrate. The specific structure and function of the neural network circuit can be referred to the neural network circuit described in the Figures 2 to 8 The neural network circuit in the corresponding embodiment described in the embodiment of the disclosure, for the sake of brevity of the specification, will not be described here.

[0098] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects of the disclosure disclosed herein. The disclosure is intended to cover any variations, uses or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure that come within known

[0099] It should be understood that the present disclosure is not limited to the precise construction that has been described above and shown in the accompanying drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the present disclosure. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A neural network circuit, characterized by, Comprise: M execution modules and a control module, wherein each of the M execution modules stores feature maps and convolution kernels corresponding to different input channels, and the feature maps and the convolution kernels in the M execution modules are stored at the same physical address, M is an integer greater than or equal to 2, each of the execution modules comprises a data calculation submodule, and the data calculation submodule comprises L calculation components; L is an integer greater than or equal to 2; and wherein, The control module is configured to: Receive a first instruction sent by an external controller; and According to the first instruction, transmit a control signal to the M execution modules, wherein the control signal is used to indicate a target physical address in the M execution modules; and Each of the M execution modules is configured to: In response to the received control signal, read the stored first feature map and first convolution kernel from the target physical address; wherein the first feature map comprises a plurality of feature map elements, and the first convolution kernel comprises K convolution kernel elements; K is an integer greater than or equal to 2; and Based on the first convolution kernel, perform convolution operation on the first feature map; Wherein, the convolution operation based on the first convolution kernel comprises simultaneously inputting a jth convolution kernel element in the K convolution kernel elements into N calculation components, and inputting N feature map elements corresponding to the jth convolution kernel element into the N calculation components respectively, so that the N calculation components simultaneously complete the convolution operation of the jth convolution kernel element and the N feature map elements, wherein the N feature map elements are contained in the plurality of feature map elements, N is an integer greater than or equal to 2, and N is less than or equal to L.

2. The neural network circuit of claim 1, wherein, The convolution operation comprises at least one of the following: pooling convolution and depth convolution.

3. The neural network circuit of claim 1, wherein, Each of the M execution modules further comprises a data control submodule and a data storage submodule, wherein the first feature map and the first convolution kernel are stored in the data storage submodule, and wherein, The data control submodule is configured to: Receive the control signal transmitted by the control module; According to the control signal, obtain the target physical address; According to the target physical address, read the first feature map and the first convolution kernel from the data storage submodule; and Input the read first feature map and the first convolution kernel into the data calculation submodule; and The data calculation submodule is configured to: Receive the first feature map and the first convolution kernel input by the data control submodule; and Based on the first convolution kernel, perform convolution operation on the first feature map.

4. The neural network circuit of claim 3, wherein, The data calculation submodule is further configured to output the first feature map after convolution to the data control submodule; The data control submodule is further configured to write the first feature map after convolution into the data storage submodule.

5. The neural network circuit of claim 3, wherein, Each of the M execution modules is further configured to perform a target operation on the first feature map and / or the first convolution kernel, and perform a convolution operation based on the operated first convolution kernel and / or the first feature map.

6. The neural network circuit of claim 5, wherein, Each of the M execution modules is further configured to perform at least one of: performing a convolution operation on the first feature map based on the operated first convolution kernel; performing a convolution operation on the operated first feature map based on the first convolution kernel; and performing a convolution operation on the operated first feature map based on the operated first convolution kernel.

7. The neural network circuit of claim 5, wherein, The target operation comprises at least one of: an interpolation operation on a feature map, a padding operation on a feature map, a dilation operation on a convolution kernel, a rotation operation on a convolution kernel.

8. The neural network circuit of claim 5, wherein, The control module is further configured to input an operand required by the target operation to each of the M execution modules.

9. The neural network circuit of claim 5, wherein, The control module is further configured to instruct each of the M execution modules to acquire the operand required by the target operation in the data storage submodule.

10. A neural network chip, comprising: comprising: a substrate; a neural network circuit as claimed in any one of claims 1 to 9 disposed on the substrate.

Images