A neural network acceleration method, accelerator, electronic device and storage medium

Abstract

Embodiments of the application disclose a neural network acceleration method, an accelerator, an electronic device and a storage medium. The method is applied to a convolution processing unit array including M rows and N columns of convolution processing units. The convolution processing unit includes X multiplication-addition units, each multiplication-addition unit includes Y multipliers, one adder and one controller. The outputs of the Y multipliers are connected to the Y inputs of the adder. The controller determines whether all input channels are not traversed, and sends the accumulated result to the Y+1th input of the adder. The controller determines whether all input channels are traversed, and controls the output of the accumulated result. In this way, the multiplication-addition unit controls the multiplication and accumulation operation of all input channels in segments, so that the final output of the multiplication-addition unit is the result of simultaneous accumulation in the plane direction and the depth direction of the convolution kernel, reducing the influence of the size and depth of the convolution kernel on the utilization rate of the multiplier. And the parallel convolution operation is realized by the multi-row convolution processing unit, and the convolution operation speed is accelerated.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to a neural network acceleration method, accelerator, electronic device and storage medium. Background Technology

[0002] The basic operation of convolution is multiplication and accumulation. Convolution can be implemented by a processing element (PE) array, which consists of m rows and n columns of PE units. Each PE is cascaded horizontally to transfer feature maps, and each PE is cascaded vertically to perform multiplication and accumulation operations.

[0003] For convolutional neural networks (CNNs), the input feature map data is a 3D data array of width * height * channels, defined as IW * IH * IC. In actual convolution calculations, the depth direction, i.e., the IC direction, also needs to be considered. The convolution calculation requires accumulating the results of each planar convolution window in the depth direction. Each output pixel is ultimately the result of simultaneous accumulation in both the planar and depth directions of the convolution kernel. In practical artificial intelligence networks, the size and depth of the convolution kernels in each convolutional layer are different. The planar size of the convolution kernels is generally composed of classic values ​​such as 1*1, 3*3, and 5*5, while the depth varies greatly. Depth has a significant impact on the utilization rate of multipliers in the PE (Precision Array), making it difficult for multipliers in the PE to achieve high utilization. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application aim to provide a neural network acceleration method, an accelerator, an electronic device, and a storage medium.

[0005] The technical solution of this application is implemented as follows:

[0006] Firstly, a neural network acceleration method is provided, the method comprising:

[0007] Obtain the input feature map data;

[0008] The input feature map data is input into a convolution processing unit array for convolution operation to obtain the output feature map data;

[0009] The convolution processing unit array includes M rows and N columns of convolution processing units. Each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers, one adder, and one controller.

[0010] The multiplier takes feature map data and weight data as inputs to its two input terminals, respectively. The output terminals of the Y multipliers are connected to the Y input terminals of the adder. The adder is used to accumulate all the input data and output the accumulated result.

[0011] If the controller determines that all input channels have not been traversed, it will send the accumulated result of the adder to the (Y+1)th input terminal of the adder.

[0012] The controller determines that all input channels have been traversed and then controls the output of the accumulated result of the adder.

[0013] Secondly, a neural network accelerator is provided, including:

[0014] An array of convolutional processing units is used to perform convolution operations on the input feature map data to obtain the output feature map data.

[0015] The convolution processing unit array includes M rows and N columns of convolution processing units. Each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers, one adder, and one controller.

[0016] The multiplier takes feature map data and weight data as inputs to its two input terminals, respectively. The output terminals of the Y multipliers are connected to the Y input terminals of the adder. The adder is used to accumulate all the input data and output the accumulated result.

[0017] The controller determines that all channels of the input feature map data have not been traversed, and sends the accumulation result of the adder to the (Y+1)th input terminal of the adder;

[0018] The controller determines that all channels of the input feature map data have been traversed, and then controls the output of the accumulated result of the adder.

[0019] Thirdly, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor.

[0020] The memory is used to store computer programs, and the processor is configured to execute the steps of the aforementioned method when running the computer programs.

[0021] Fourthly, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.

[0022] This application provides a neural network acceleration method, accelerator, electronic device, and storage medium. The method utilizes a convolutional processing unit array comprising M rows and N columns. Each convolutional processing unit includes X multiply-accumulate units, each containing Y multipliers, one adder, and a controller. The outputs of the Y multipliers are connected to the Y inputs of the adder. The controller, upon determining that all input channels have not been fully traversed, sends the accumulated result to the (Y+1)th input of the adder. Upon determining that all input channels have been fully traversed, the controller controls the output of the accumulated result. This allows the multiply-accumulate units to perform segmented multiplication and accumulation operations on all input channels, ensuring that the final output of the multiply-accumulate unit is the result of simultaneous accumulation in both the planar and depth directions of the convolution kernel, reducing the impact of kernel size and depth on multiplier utilization. Furthermore, the use of multiple rows of convolutional processing units enables parallel convolution operations, accelerating the convolution operation speed. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the first process of the neural network acceleration method in the embodiments of this application;

[0024] Figure 2 This is a schematic diagram of the first component structure of the convolution processing unit array in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of the first component structure of the convolution processing unit in an embodiment of this application;

[0026] Figure 4 This is a schematic diagram of the second component structure of the convolution processing unit array in an embodiment of this application;

[0027] Figure 5 This is a schematic diagram of the second component structure of the convolution processing unit in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the second process of the neural network acceleration method in the embodiments of this application;

[0029] Figure 7 This is a schematic diagram of the feature map input sequence in the embodiments of this application;

[0030] Figure 8 This is a schematic diagram illustrating the output order of the feature maps in the embodiments of this application;

[0031] Figure 9 This is a flowchart illustrating the process of the convolution processing unit implementing 3*3 convolution operation in an embodiment of this application.

[0032] Figure 10 This is a schematic diagram of the third process of the neural network acceleration method in the embodiments of this application;

[0033] Figure 11This is a flowchart illustrating the process of a convolution processing unit implementing a 1x1 convolution operation in an embodiment of this application.

[0034] Figure 12 This is a flowchart illustrating the process of the convolution processing unit implementing 2*2 convolution operation in an embodiment of this application.

[0035] Figure 13 This is a schematic diagram of the composition structure of the electronic device in the embodiments of this application. Detailed Implementation

[0036] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0037] Currently, the PE array consists of m rows and n columns of PE units. Each PE is cascaded horizontally for feature map transmission and vertically for multiply-accumulate operations. The utilization rate of multipliers in the PE array depends on the kernel size and depth. For example, assuming the internal dimensions of a PE unit are x=8 and y=8, meaning each PE unit contains 8 multiply-accumulate units horizontally, each containing 8 multipliers, for a total of 64 multipliers, and the PE array dimensions are m=8 and n=8, there are 64 multipliers vertically, which are cascaded for accumulation operations, and 64 columns horizontally, corresponding to 64 convolutional kernels. Therefore, if the kernel depth or the number of convolutional kernels is not a multiple of 64, many multipliers in the PE array will be wasted, making it difficult to achieve high multiplier utilization.

[0038] In view of this, this application provides a neural network acceleration method applied to a novel PE array structure. Figure 1 This is a schematic diagram of the first process of the neural network acceleration method in the embodiments of this application, as shown below. Figure 1 As shown, the method may specifically include:

[0039] Step 101: Obtain input feature map data;

[0040] Step 102: Input the input feature map data into the convolution processing unit array for convolution operation to obtain the output feature map data;

[0041] For convolutional neural networks, the input feature map data is a three-dimensional data array with width * height * channels, defined as IW * IH * IC. The input feature map data can be single-channel, three-channel, four-channel, etc.

[0042] This application provides a hardware implementation structure for a convolutional neural network, namely a convolutional processing unit array, or "PE array" for short. For example, the input feature map data is mapped sequentially to the input of the multiplier according to the channel order, or the input feature map data is segmented along the depth direction (i.e., the channel direction) to form at least one set of data, and each set of data is mapped to the input of the multiplier.

[0043] Figure 2 This is a schematic diagram of the first component structure of the convolution processing unit array in an embodiment of this application. The PE array includes M rows and N columns of convolution processing units (PE for short), where M and N are positive integers.

[0044] Figure 3 This is a schematic diagram of the first component structure of the convolution processing unit in an embodiment of this application, as shown below. Figure 3 As shown, each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers 301, one adder 302 and one controller 303;

[0045] The two input terminals of the multiplier 301 are respectively input to feature map data (F) and weight data (W). The output terminals of the Y multipliers 301 are connected to the Y input terminals of the adder 302. The adder 302 is used to accumulate all the input data and output the accumulated result.

[0046] The controller 303 determines that all input channels have not been traversed, and sends the accumulated result of the adder 302 to the (Y+1)th input terminal of the adder 302;

[0047] The controller 303 determines that all input channels have been traversed and then controls the output of the accumulated result of the adder 302.

[0048] Here, the number of convolution processing units, the number of multiply-accumulate units within each convolution processing unit, and the number of multipliers can all be flexibly set according to the actual convolution operation requirements. It is also possible to set the most commonly used 3x3 convolution kernel and then reuse it for other sizes of convolution kernels.

[0049] The aforementioned convolutional processing unit array supports convolution operations with arbitrary kernels and has no special requirements on kernel depth. In particular, when the kernel size equals the number of multipliers in the multiply-accumulate unit, there are no requirements on kernel depth. When the kernel size is not equal to the number of multipliers in the multiply-accumulate unit, if the kernel depth is an integer multiple of the number of multipliers, the multiplier utilization can reach 100%. If the kernel depth is not an integer multiple of the number of multipliers, the impact on multiplier utilization is minimal; overall, the multiplier utilization is high.

[0050] For example, all multiply-accumulate units in a convolutional processing unit share the weight data of a single convolutional kernel. That is, a convolutional processing unit performs convolution operations on all channels of the feature map with a single convolutional kernel to obtain a single feature map. When M and N are integers greater than 1, each convolutional processing unit in the same row has a different convolutional kernel, while each convolutional processing unit in the same column has the same convolutional kernel. Here, each column of convolutional processing units uses the same convolutional kernel, and M rows of convolutional processing units perform parallel convolution operations on the input feature map data. Each row of convolutional processing units performs convolution operations on the input feature map data with N convolutional kernels, outputting N feature maps.

[0051] For example, the convolution processing unit also includes a memory for pre-storing the weight data of the convolution kernel. Once the convolution kernel of each convolution processing unit is determined, it is pre-stored in the memory of each unit, and the convolution operation of all feature map data is achieved by updating the feature map data. For example, the feature map data can be input laterally in a pulsed manner or in a broadcast manner.

[0052] In other embodiments, the memory is used to pre-store feature map data. That is, the feature map data can also be fixed in each PE, and convolution operations can be performed by updating the weight data. For example, the weight data can be input laterally in a pulsed manner or via broadcasting. Optionally, updating the weight data can reduce the number of convolution processing units and simplify the hardware structure when there are many convolution kernels.

[0053] In other embodiments, the convolution kernels of each convolutional processing unit in the same row are different, while the convolution kernels of each convolutional processing unit in the same column are the same. That is, the convolutional processing unit array performs convolution operations on the input feature map data with M*N convolution kernels, outputting M*N feature maps. For example, the input method for each PE feature map data can be either a pulsed horizontal input or a broadcast input.

[0054] For example, in some embodiments, the convolution processing unit array includes M multiplexers, the N inputs of which are connected to the N outputs of a row of convolution processing units, for selecting one output in each clock cycle.

[0055] In practical applications, the number of input terminals of a multiplexer can be equal to N or greater than N. When the number of input terminals of a multiplexer is greater than N, the extra input terminals can be reserved for expanding the PE array.

[0056] For example, in some other embodiments, the convolution processing unit may also directly output the convolution processing result without setting a multiplexer.

[0057] The following provides a further example illustrating the convolutional processing unit array. Figure 4 This is a schematic diagram of the second component structure of the convolution processing unit array in an embodiment of this application, as shown below. Figure 4 As shown, the convolutional processing unit array includes 4 rows and 4 columns of PEs, namely PE0, PE1 to PE15. The convolutional kernels of each convolutional processing unit in the same row are different, namely K0, K1, K2 and K3 respectively. The convolutional kernels of each convolutional processing unit in the same column are the same.

[0058] The four rows of PEs perform parallel convolution operations on the four convolution windows in the input feature map. The first row of PEs performs the convolution operation between search window 0 and the four convolution kernels. The second row of PEs performs the convolution operation between search window 1 and the four convolution kernels. The third row of PEs performs the convolution operation between search window 2 and the four convolution kernels. The parallel processing of the four search windows ends. The search window positions in the input feature map data are moved to obtain four new search windows. This process continues until the entire input feature map data has undergone convolution operations, resulting in four output feature map data after four types of convolution operations.

[0059] In practical applications, four rows of PEs process four different search windows, each projected onto the feature map plane at different coordinate positions. Figure 4 As shown, the four search windows can be four windows vertically in the feature map or four windows horizontally. In fact, four windows horizontally are also possible. Depending on the actual convolution operation requirements, the four search windows may or may not overlap.

[0060] Taking the most commonly used 3*3 convolution kernel as an example, each PE can process a search window of size 4*10. There is half overlap between two adjacent search windows. For a search window of input feature map data, its size is 4*10*IC.

[0061] Each PE structure, such as Figure 5As shown, this includes 16 columns of multiply-accumulate units. In 3x3 convolution mode, each multiply-accumulate unit has 9 multipliers, and each column can process 9 multiplication operations in parallel, corresponding to the multiplication of a 3x3 convolution window with the weights. Different multiply-accumulate units process different 3x3 convolution windows, so 16 convolution windows can be processed simultaneously, generating 16 partial sums. Each column of multiply-accumulate units can perform a self-accumulation operation to obtain the partial sum. For example, if the feature map has 1 input channel, the result generated by each column of multiply-accumulate units can be output as an output point. However, if the input channels are greater than 1, the first result generated by each column of multiply-accumulate units is only a partial sum. It is necessary to accumulate the multiply-accumulate results of all channels until all input channels have been traversed before a final output point can be generated. Figure 5 The controller determines whether the output of the multiply-accumulate unit continues to participate in the next accumulation or directly outputs the convolution result of the search window based on whether the input channel has been traversed. Therefore, in 3*3 convolution mode, each PE can perform convolution operations on 16 convolution window plane data on a single input channel at a time, generating 16 convolution results.

[0062] Here, the convolution processing unit performs convolution operations on the planar data of the search window in each clock cycle. The internal structure of the convolution processing unit is related to the kernel size; that is, the kernel size equals the number of multipliers in the multiply-accumulate unit. For example, when the kernel size is 2x2, the multiply-accumulate unit includes 4 multipliers; when the kernel size is 3x3, the multiply-accumulate unit includes 9 multipliers; and when the kernel size is 5x5, the multiply-accumulate unit includes 25 multipliers.

[0063] For example, such as Figure 6 As shown, neural network acceleration methods include:

[0064] Step 601: Obtain M search windows from the input feature map data;

[0065] Here, the search window is a 3D data array with width * height * channels. The feature map data within the search window can be understood as a portion of the feature map data from the input feature map data. This feature map data within the search window can be mapped horizontally to the convolutional processing unit for convolution operations; that is, the convolutional processing unit processes the planar data of the search window. The number of channels in the search window is equal to the number of channels in the input feature map data. When multiple channels are present, the convolutional processing unit processes one channel of planar data per clock cycle.

[0066] M-row convolution processing units can process M search windows in parallel. The convolution operation process is the same for each row. The following example illustrates the convolution operation process of a single-row convolution processing unit.

[0067] Step 602: In each clock cycle, the M search windows are input in parallel into the M rows of convolution processing units according to the channel order;

[0068] Step 603: The convolution processing unit obtains X convolution windows of the search window and flattens and maps the X convolution windows to the input of the multipliers of the X multiply-accumulate units;

[0069] Step 604: In each clock cycle, the first convolutional processing unit in each row will laterally pass the input feature map data to the next convolutional processing unit until it is passed laterally to the last convolutional processing unit.

[0070] like Figure 7 As shown, the input feature map size is IW*IH*IC. Assuming the number of channels in the input feature map is 4, one channel of planar data is input into the PE every clock cycle.

[0071] like Figure 8 As shown, an example illustrates the output order of feature maps during convolution operations in a row of PEs. Each PE unit contains 16 multiply-accumulate units that share a set of weight data. Different PEs use different weight data: PE0 uses kernel K0 as its weight data, PE1 uses K1, and so on. Each PE processes 16 convolution operations across 3x3 windows, resulting in 16 outputs. Assuming the input feature map has 4 channels, since each PE (e.g., ...) ... Figure 5 As shown, each convolutional processing unit (PE) can only process planar data from one channel at a time. Each PE requires four time cycles to generate a complete output point. The depth of the output point can be considered as the maturity of the output window result. After each PE processes 16 convolutional windows of the feature map, it temporarily stores a portion of the data and laterally propagates the data from the 16 convolutional windows to the next PE. Figure 8 As shown, PE0 generates a complete output window (output window size is 8*2) at time t0, corresponding to the convolution result of the search window and K0. PE1 generates a complete output window at time t1, corresponding to the convolution result of the search window and K1, and so on. Therefore, for the four output window data of the four periods from t0 to t3, a multiplexer MUX is needed to select the output of one row of PE in each period. At time t0, the convolution result of PE0 is selected for output; at time t1, the convolution result of PE1 is selected for output, and so on. The position of the multiplexer MUX is as follows: Figure 4 As shown.

[0072] In other words, for a single PE, each time period receives data from different channels of the same search window. For example, for a PE, the first time period processes the data of channel 0 of the search window, and the controller chooses to continue accumulating instead of outputting. The second time period processes the data of channel 1 of the search window, and the controller chooses to continue accumulating instead of outputting, until the PE has processed all channels of the current search window, and the controller outputs the convolution results of all channels.

[0073] like Figure 9 As shown, the convolution operation process performed by the PE is illustrated. Each PE can process the data of one search window of the input feature map simultaneously. The search window can contain 16 3x3 convolution windows. The scheduler reads the data of the 16 3x3 convolution windows from memory and performs a flattening operation, mapping the 16 convolution window data to 16 multiply-accumulate units in the PE. Each PE pre-stores the weight data required for the convolution operation, and each PE shares the weight data of a weight kernel, that is... Figure 9 After W0 and PE receive the convolution window data, they can perform multiplication and accumulation operations. According to the characteristics of convolution operations, a 4*10 search window can generate 2*8 output window data.

[0074] In practical applications, if the internal structure of the convolution processing unit is independent of the convolution kernel size, for example using... Figure 5 The PE unit implements 1x1, 2x2, and 5x5 convolution operations. Unlike the 3x3 unit, which flattens and maps the convolution window to the multiplier input, this unit needs to flatten in the depth direction. That is, the PE performs convolution operations on the feature map data of X convolution windows along the depth direction in each clock cycle. In this case, the internal structure of the convolution processing unit is related to the kernel depth. If the kernel depth is an integer multiple of the number of multipliers in the multiply-accumulate unit, the multiplier utilization can reach 100%. If it is not an integer multiple, the multipliers will not be fully utilized only when processing the last segment of feature map data, but overall, the multiplier utilization is still relatively high.

[0075] For example, such as Figure 10 As shown, neural network acceleration methods include:

[0076] Step 1001: Obtain M search windows from the input feature map data;

[0077] Here, the search window is a 3D data array with width * height * channels. The feature map data within the search window can be understood as a portion of the feature map data from the input feature map data. This feature map data within the search window can be mapped along the depth direction to the convolutional processing unit for convolution operations. In other words, the convolutional processing unit processes a segment of data within the search window along the depth direction. The data within the search window is segmented along the depth direction, and the convolutional processing unit processes one segment of data per clock cycle.

[0078] M-row convolution processing units can process M search windows in parallel. The convolution operation process is the same for each row. The following example illustrates the convolution operation process of a single-row convolution processing unit.

[0079] Step 1002: In each clock cycle, take Y points along the depth direction in the X convolutional windows of the search window to form X sets of feature map data;

[0080] Step 1003: Input the X sets of feature map data corresponding to each of the M search windows into the M rows of convolutional processing units in parallel;

[0081] Each convolutional processing unit obtains X sets of feature map data, where X multiply-accumulate units process the X sets of feature map data in parallel, and Y points of a set of feature map data are mapped to the input of Y multipliers of a multiply-accumulate unit.

[0082] Step 1004: The convolution processing unit maps X sets of feature map data to the input of the multiplier of the X multiply-accumulate unit;

[0083] Step 1005: In each clock cycle, the first convolutional processing unit in each row will laterally pass the input feature map data to the next convolutional processing unit until it is passed laterally to the last convolutional processing unit.

[0084] like Figure 11As shown, when implementing a 1x1 convolution operation, each search window includes 16 1x1 convolution windows, which need to be flattened in the depth direction. The black points in the input feature map (which can be understood as the first 1x1 convolution window) represent the set of a point on the XY plane across all channels. Every nine points along the depth direction form a group of feature map data, which is mapped to the input of a multiplier in a multiply-accumulate unit of the PE (Preprocessor Entity). That is, the black point is assigned to the multiply-accumulate unit within the black box, the second gray point (which can be understood as the second 1x1 convolution window) is assigned to the multiply-accumulate unit within the gray box, and so on, until the sixteenth point (which can be understood as the sixteenth 1x1 convolution window) is assigned to the sixteenth multiply-accumulate unit, generating a partial sum or the final output point. Therefore, for each PE, 16 partial sums can be generated at once. A convolution kernel depth that is an integer multiple of 9 allows the multiplier to achieve 100% utilization. If the depth is not an integer multiple, the multiplier will not be fully utilized only when processing the last segment of feature map data, but overall, the multiplier utilization remains high.

[0085] For other kernel sizes, let's take a 2x2 convolution operation as an example, such as... Figure 12 As shown, to implement a 2x2 convolution operation, each search window consists of 16 2x2 convolution windows, which need to be flattened in the depth direction. There are 16 2x2 convolution windows in the search window, each labeled 1, 2, 3, and 4 to represent the order in which the inputs are fed to the PE (Process Execution Unit). For example, in the first clock cycle, the first data point (number 1) of the first convolution window (channels 0-8) is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate unit in the black box for accumulation. Then, the first data point (number 1) of the second convolution window (channels 0-8) is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate unit in the gray box, and so on.

[0086] If the input channels are 9 (i.e., the kernel depth is 9), in the second clock cycle, data point number 2 from the first convolution window is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate units in the black box. Similarly, data point number 2 from the second convolution window is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate units in the gray box. Simultaneously, all data points number 1 from channels 0-8 are fed into the next PE for processing, and so on. By the fourth clock cycle, the convolution processing of 9 channels in the 16 convolution windows within the PE search window is complete.

[0087] If the input channels are greater than 9 (i.e., the kernel depth is greater than 9), in the second clock cycle, the first data point 1 of the first convolution window of channels 9-17 is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate unit in the black box. At the same time, the first data point 1 of the second convolution window of channels 9-17 is retrieved along the depth direction, totaling 9 data points, and assigned to the multiply-accumulate unit in the gray box. Simultaneously, all the first data points 1 of channels 0-8 are sent to the next PE for processing, and so on, until all input channels of the first data point 1 have been traversed.

[0088] It should be noted that when the controller has completed traversing all input channels at each point, it outputs the convolution result of each point separately, or when it has completed traversing all input channels at all four points of the convolution window, it outputs the convolution results of all four points together.

[0089] For other convolutional kernel sizes, such as 5x5 and 7x7, the operation is similar to that of 2x2. The difference is that in 2x2 mode, it takes four steps to generate the partial sum of 16 convolutional windows, while for 5x5 convolution it takes 25 steps, and for 7x7 convolution it takes 49 steps.

[0090] The above-mentioned convolution processing unit array achieves parallel processing of convolution operations by setting up M rows of convolution processing units. The optimized convolution processing unit structure eliminates the data dependency between each convolution processing unit, reduces the impact of the convolution algorithm on the multiplier utilization, and enables the convolution processing array to support convolution operations with arbitrary convolution kernels without special requirements on the depth of the convolution kernel. It is highly flexible and scalable.

[0091] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a neural network accelerator, which includes:

[0092] A convolutional processing unit array is used to perform convolution operations on input feature map data to obtain output feature map data; the convolutional processing unit array is any one of the convolutional processing unit arrays described in the embodiments of this application.

[0093] The convolution processing unit array includes M rows and N columns of convolution processing units. Each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers, one adder, and one controller.

[0094] The multiplier takes feature map data and weight data as inputs to its two input terminals, respectively. The output terminals of the Y multipliers are connected to the Y input terminals of the adder. The adder is used to accumulate all the input data and output the accumulated result.

[0095] The controller determines that all channels of the input feature map data have not been traversed, and sends the accumulation result of the adder to the (Y+1)th input terminal of the adder;

[0096] The controller determines that all channels of the input feature map data have been traversed, and then controls the output of the accumulated result of the adder.

[0097] In some embodiments, all multiply-accumulate units in a convolution processing unit share the weight data of a single convolution kernel.

[0098] In some embodiments, when M and N are integers greater than 1, the convolution kernels of each convolution processing unit in the same row are different, and the convolution kernels of each convolution processing unit in the same column are the same.

[0099] In some embodiments, the convolutional processing unit array is used to obtain M search windows from the input feature map data; in each clock cycle, the M search windows are input in parallel to the M rows of convolutional processing units in channel order; the convolutional processing units obtain X convolutional windows of the search windows, and flatten and map the X convolutional windows to the input of the multipliers of the X multiply-accumulate units.

[0100] In some embodiments, the convolutional processing unit array is used to obtain M search windows from the input feature map data; in each clock cycle, Y points are taken along the depth direction in the X convolutional windows of the search windows to form X sets of feature map data; the X sets of feature map data corresponding to each of the M search windows are input in parallel into M rows of convolutional processing units; the convolutional processing units map the X sets of feature map data to the input of the multiplier of the X multiply-accumulate units.

[0101] In some embodiments, when N is an integer greater than 1, two adjacent convolution processing units in the same row are connected.

[0102] The array of convolutional processing units is further used in each clock cycle, where the first convolutional processing unit in each row laterally passes the input feature map data to the next convolutional processing unit until the data is passed laterally to the last convolutional processing unit.

[0103] In some embodiments, the convolution processing unit array includes M multiplexers, the N inputs of which are connected to the N outputs of a row of convolution processing units, for selecting one output in each clock cycle.

[0104] In some embodiments, the convolution processing unit further includes a memory for pre-storing weight data of the convolution kernel or pre-storing feature map data.

[0105] In practical applications, a neural network accelerator can be a convolution processing chip, which can also be called a system-on-a-chip, system-on-a-chip, chip system, or system-on-a-chip, etc.

[0106] The neural network accelerator can implement convolutional processing unit arrays and neural network acceleration methods through a combination of software and hardware. Furthermore, the technical effects of each technical solution of this neural network accelerator can be referenced from the technical effects of corresponding technical solutions in convolutional processing methods; therefore, this application will not elaborate further on these aspects.

[0107] This application also provides an electronic device, such as... Figure 13 As shown, the electronic device 130 includes: a processor 1301 and a memory 1302 configured to store computer programs capable of running on the processor.

[0108] The memory 1302 is used to store computer programs, the processor 1301 and the memory 1302 configured to store computer programs that can run on the processor.

[0109] Wherein, the processor 1301 is configured to execute the method steps in the foregoing embodiments when running a computer program. Of course, in practical applications, such as... Figure 13 As shown, the various components in this electronic device are coupled together via a bus system 1303. It is understood that the bus system 1303 is used to enable communication between these components. In addition to a data bus, the bus system 1303 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 13 The general designated all buses as Bus System 1303.

[0110] In practical applications, the electronic device further includes the aforementioned neural network accelerator, with the processor controlling the neural network accelerator to perform convolution operations. Alternatively, the processor may include the aforementioned neural network accelerator, with the processor itself implementing the convolution operation function of the neural network accelerator.

[0111] In practical applications, the aforementioned processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic devices used to implement the functions of the aforementioned processor can also be other types, and the embodiments of this application do not specifically limit this.

[0112] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0113] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to perform the steps of the aforementioned method.

[0114] This application also provides a computer program product, including computer program instructions.

[0115] Optionally, the computer program product can be applied to the electronic device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.

[0116] This application also provides a computer program.

[0117] Optionally, the computer program can be applied to the electronic device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the electronic device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0118] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.

[0119] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.

[0120] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0122] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0124] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A neural network acceleration method, characterized in that, The method includes: Obtain the input feature map data; The input feature map data is input into a convolution processing unit array for convolution operation to obtain the output feature map data; The convolution processing unit array includes M rows and N columns of convolution processing units. Each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers, one adder, and one controller. The multiplier takes feature map data and weight data as inputs to its two input terminals, respectively. The output terminals of the Y multipliers are connected to the Y input terminals of the adder. The adder is used to accumulate all the input data and output the accumulated result. If the controller determines that all input channels have not been traversed, it will send the accumulated result of the adder to the (Y+1)th input terminal of the adder. The controller determines that all input channels have been traversed and then controls the output of the accumulated result of the adder. Wherein, when the convolution kernel size is equal to the number of multipliers Y in the multiply-accumulate unit, the step of inputting the input feature map data into the convolution processing unit array for convolution operation includes: M search windows are obtained from the input feature map data; In each clock cycle, the M search windows are input in parallel to the M rows of convolution processing units in channel order, and the convolution processing units process planar data of one channel in each clock cycle. The convolution processing unit obtains X convolution windows from the search window and flattens and maps the X convolution windows to the input of the multipliers of the X multiply-accumulate units; Alternatively, when the convolution kernel size is not equal to the number of multipliers Y in the multiply-accumulate unit, the step of inputting the input feature map data into the convolution processing unit array for convolution operation includes: M search windows are obtained from the input feature map data; In each clock cycle, Y points are taken along the depth direction in the X convolutional windows of the search window to form X sets of feature map data; The X sets of feature map data corresponding to each of the M search windows are input in parallel into the M rows of convolutional processing units; The convolution processing unit maps X sets of feature map data to the input of the multipliers of X multiply-accumulate units.

2. The method according to claim 1, characterized in that, In a convolutional processing unit, all multiply-accumulate units share the weight data of a single convolutional kernel.

3. The method according to claim 2, characterized in that, When M and N are integers greater than 1, the convolution kernels of each convolution processing unit in the same row are different, while the convolution kernels of each convolution processing unit in the same column are the same.

4. The method according to claim 1, characterized in that, When N is an integer greater than 1, two adjacent convolution processing units in the same row are connected. The step of inputting the input feature map data into the convolution processing unit array for convolution operation further includes: In each clock cycle, the first convolutional processing unit in each row passes the input feature map data laterally to the next convolutional processing unit until it is passed laterally to the last convolutional processing unit.

5. The method according to claim 1, characterized in that, The convolution processing unit array includes M multiplexers, and the N inputs of the multiplexers are connected to the N outputs of a row of convolution processing units to select one output in each clock cycle.

6. The method according to claim 1, characterized in that, The convolution processing unit also includes a memory. The memory is used to pre-store the weight data of the convolution kernel or to pre-store the feature map data.

7. A neural network accelerator, characterized in that, include: An array of convolutional processing units is used to perform convolution operations on the input feature map data to obtain the output feature map data. The convolution processing unit array includes M rows and N columns of convolution processing units. Each convolution processing unit includes X multiply-accumulate units, and each multiply-accumulate unit includes Y multipliers, one adder, and one controller. The multiplier takes feature map data and weight data as inputs to its two input terminals, respectively. The output terminals of the Y multipliers are connected to the Y input terminals of the adder. The adder is used to accumulate all the input data and output the accumulated result. The controller determines that all channels of the input feature map data have not been traversed, and sends the accumulation result of the adder to the (Y+1)th input terminal of the adder; The controller determines that all channels of the input feature map data have been traversed, and then controls the output of the accumulated result of the adder. When the kernel size is equal to the number of multipliers Y in the multiply-accumulate unit, the convolution processing unit array is used to obtain M search windows from the input feature map data; in each clock cycle, the M search windows are input in parallel to the M rows of convolution processing units in channel order, and the convolution processing unit processes the planar data of one channel in each clock cycle; the convolution processing unit obtains X convolution windows from the search windows, and flattens and maps the X convolution windows to the input of the multipliers of the X multiply-accumulate units; or, When the kernel size is not equal to the number Y of multipliers in the multiply-accumulate unit, the convolution processing unit array is used to obtain M search windows from the input feature map data; in each clock cycle, Y points are taken along the depth direction in the X convolution windows of the search windows to form X sets of feature map data; the X sets of feature map data corresponding to each of the M search windows are input in parallel into the M rows of convolution processing units; the convolution processing units map the X sets of feature map data to the input of the multipliers of the X multiply-accumulate units.

8. The neural network accelerator according to claim 7, characterized in that, In a convolutional processing unit, all multiply-accumulate units share the weight data of a single convolutional kernel.

9. The neural network accelerator according to claim 8, characterized in that, When M and N are integers greater than 1, the convolution kernels of each convolution processing unit in the same row are different, while the convolution kernels of each convolution processing unit in the same column are the same.

10. An electronic device, characterized in that, include: The processor and memory configured to store computer programs that can run on the processor. The memory is used to store a computer program, and the processor is configured to execute the steps of the method according to any one of claims 1 to 6 when running the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images