A configurable deep separable convolution hardware acceleration system and method

Abstract

The application discloses a configurable deep separable convolution hardware acceleration system and method, comprising: a feature map storage buffer and a weight storage buffer, which are used for storing the input and weight of a current convolution network layer; a configurable data flow convolution module, which is used for deep separable convolution operation on the input, and each round of data of part channels of the input feature map is subjected to corresponding convolution operation; a configurable addition tree module, which is used for accumulating the convolution result, and the convolution result of each round is subjected to corresponding accumulation; an intermediate result accumulation module, which is used for accumulating the point-by-point convolution result of part channels of each round with the point-by-point convolution result of part channels of the previous round until all input channels are completely accumulated to output the result; and an accumulated result processing module, which is used for processing the convolution accumulation result to obtain the output of the current convolution layer. The configurable main calculation module greatly reduces the area and resource overhead of the design.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network hardware acceleration, and in particular relates to a configurable depthwise separable convolution hardware acceleration system and method. Background Art

[0002] With the continuous advancement of neural network technology, it is becoming increasingly common in multiple practical application areas, such as image recognition, speech processing, and autonomous driving. However, as the number of network layers increases, the number of model parameters also surges, which brings a series of challenges to the deployment of neural networks on mobile devices. The huge computational requirements lead to a significant increase in the latency of neural networks during inference, which is unacceptable in some application scenarios. Therefore, the neural networks deployed on mobile devices are often lightweight networks. These network models contain a large number of depthwise separable convolution operations. However, current acceleration solutions for depthwise separable convolution generally lack flexibility, and no universal acceleration strategy has been developed for the pointwise convolution and depthwise convolution operations in depthwise separable convolution. This leads to insufficient utilization of hardware modules when performing depthwise separable convolution, thereby unnecessarily increasing the consumption of hardware resources.

[0003] Existing solutions typically use different acceleration strategies for different types of convolution operations in depthwise separable convolution. Dedicated modules are often designed to handle the feature map data flow and convolution multiplication calculations for point-by-point and depthwise convolution. The addition tree module after the convolution multiplication calculation is usually configured with a 9-input addition tree and an addition tree with the same number of parallel input channels, respectively handling the accumulation of depthwise convolution in the two-dimensional plane and the accumulation of point-by-point convolution in the direction of the input channels during the depthwise separable convolution operation. In short, traditional acceleration solutions are often too simplistic and can only be used for specific networks and FPGA devices. Design portability is poor, and because module parameters are not configurable, more modules need to be designed to meet computing needs, resulting in poor module reusability and increased resource and area overhead. Summary of the Invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a configurable depth-wise separable convolution hardware acceleration system and method.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a configurable depth-separable convolution hardware acceleration system, comprising:

[0007] The feature map storage buffer and weight storage buffer are used to store the first input feature map and weight parameters of the current depth-separable convolutional network layer respectively;

[0008] A configurable data flow convolution module is configured to perform a depthwise convolution operation on the first input feature map in a depthwise convolution data flow convolution mode to obtain a depthwise convolution result; in a pointwise convolution data flow convolution mode, perform a pointwise convolution operation on the data of c channels in the second input feature map of the pointwise convolution layer in each round to obtain the pointwise convolution result of this round;

[0009] A configurable addition tree module is configured to accumulate the depth convolution results in an input addition mode that matches the depth convolution to obtain a depth convolution accumulation result; and accumulate the point-by-point convolution results of each round in an input addition mode that matches the point-by-point convolution;

[0010] The intermediate result accumulation module is used to accumulate the point-by-point convolution results of each round with the point-by-point convolution results of the previous round until all the point-by-point convolution results of all rounds are accumulated;

[0011] The accumulated result processing module is used to process the accumulated result of the deep convolution to obtain the output feature map of the deep convolution layer; process the accumulated result of the point-by-point convolution results accumulated in all rounds to obtain the output feature map of the current deep separable convolutional network layer; the output feature map of the deep convolution layer is the second input feature map.

[0012] The present invention also provides a configurable depth-separable convolution hardware acceleration method, which is applied to the above-mentioned configurable depth-separable convolution hardware acceleration system, and the method includes:

[0013] Obtain the first input feature map and weight parameters of the current depthwise separable convolutional network layer respectively; each depthwise separable convolutional network layer includes a depthwise convolutional layer and a cascaded pointwise convolutional layer;

[0014] Performing a depthwise convolution operation on the first input feature map in a depthwise convolution data flow convolution mode to obtain a depthwise convolution result;

[0015] Accumulating the depth convolution results in an input addition mode matching the depth convolution to obtain a depth convolution accumulation result;

[0016] Processing the depth convolution accumulation result to obtain the output feature map of the depth convolution layer, and caching the output feature map of the depth convolution layer;

[0017] In the data flow convolution mode of point-by-point convolution, the output feature map of the depthwise convolution layer is used as the second input feature map. In each round, a point-by-point convolution operation is performed on the data of c channels in the cached second input feature map to obtain the point-by-point convolution result of this round.

[0018] accumulate the point-wise convolution result of each round in an input addition mode matched with the point-wise convolution;

[0019] accumulate the point-wise convolution result of each round in an input addition mode matched with the point-wise convolution;

[0020] accumulate the point-wise convolution result of each round in an input addition mode matched with the point-wise convolution;

[0021] Compared with the prior art, the present application has the following beneficial effects:

[0022] The present application optimizes the convolution calculation data flow convolution module, so that it can simultaneously complete point-wise convolution and depth convolution operation with different configurations, to maximize the reuse of the convolution module. A configurable addition tree module is also designed, which can simultaneously support the accumulation operation of point-wise convolution and depth convolution, and can adapt to different input channel parallelism design, making the design very flexible, with high module utilization, greatly reducing the area and resource overhead of the design, and the calculation unit can be closed, reducing the power consumption caused by unnecessary calculation.

[0023] The present application will be further described in detail below in combination with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is an exemplary structural schematic diagram of a configurable depth separable convolution hardware acceleration system provided by the embodiments of the present application;

[0025] Figure 2 is a structural schematic diagram of a configurable data flow convolution module provided by the embodiments of the present application;

[0026] Figure 3 is a principle schematic diagram of data multiplexing of depth convolution;

[0027] Figure 4 is an exemplary internal configuration schematic diagram of a configurable data flow convolution module provided by the embodiments of the present application when it works in the data flow convolution mode of depth convolution;

[0028] Figure 5 is another exemplary internal configuration schematic diagram of a configurable data flow convolution module provided by the embodiments of the present application when it works in the data flow convolution mode of depth convolution;

[0029] Figure 6 is a schematic diagram of the internal configuration of a configurable data flow convolution module provided by an embodiment of the present application when working in a point-wise convolution data flow convolution mode;

[0030] Figure 7 is an exemplary schematic diagram of the accumulation principle of deep convolution;

[0031] Figure 8 is an exemplary schematic diagram of the accumulation principle of point-wise convolution;

[0032] Figure 9 is a structural schematic diagram of a configurable adder tree module provided by an embodiment of the present application;

[0033] Figure 10 is a schematic diagram of the internal configuration of a configurable adder tree module provided by an embodiment of the present application when working in a 9-input adder mode;

[0034] Figure 11 is a schematic diagram of the internal configuration of a configurable adder tree module provided by an embodiment of the present application when working in an 8-input adder mode;

[0035] Figure 12 is a schematic diagram of the internal configuration of a configurable adder tree module provided by an embodiment of the present application when working in a 16-input adder mode;

[0036] Figure 13 is a schematic diagram of the internal configuration of a configurable adder tree module provided by an embodiment of the present application when working in a 32-input adder mode. DETAILED DESCRIPTION

[0037] The present application will be further described below in conjunction with specific embodiments, but the embodiments of the present application are not limited thereto.

[0038] The application provides a configurable deep separable convolution hardware acceleration system, and the system is a deep separable convolution hardware acceleration system based on FPGA. The system comprises a feature map storage buffer, a weight storage buffer, a configurable data flow convolution module and a configurable addition tree module, an intermediate result accumulation module and an accumulated result processing module. The feature map storage buffer and the weight storage buffer are connected with the configurable data flow convolution module to send feature map data and weight parameters to the configurable data flow convolution module. The configurable data flow convolution module is connected with the configurable addition tree module to send convolution calculation results to the configurable addition tree module for addition operation. The configurable addition tree module is connected with the intermediate result accumulation module to send data after addition calculation to the intermediate result accumulation module. The intermediate result accumulation module is also connected with the accumulated result processing module to send the calculated accumulated result to the accumulated result processing module for processing. The accumulated result processing module is also connected with the feature map storage buffer to store the processed feature map to the feature map storage buffer.

[0039] The feature map storage buffer and the weight storage buffer are used to acquire and store the input feature map and the weight parameter of the current deep separable convolution network layer respectively; each deep separable convolution network layer comprises a deep convolution layer and a cascaded point-wise convolution layer. Here, the input feature map of the current deep separable convolution network layer is referred to as the first input feature map.

[0040] The configurable data flow convolution module is used to perform deep convolution operation on the first input feature map in the data flow convolution mode of the deep convolution to obtain the deep convolution result; in the data flow convolution mode of the point-wise convolution, the data of c channels in the input feature map of the point-wise convolution layer is subjected to point-wise convolution operation every round to obtain the point-wise convolution result of this round; when the output feature map of the deep convolution layer has X channels, the total number of rounds of point-wise convolution operation is Here, since each depthwise separable convolutional network layer consists of a depthwise convolutional layer and a cascaded pointwise convolutional layer, when calculating the current depthwise separable convolutional network layer, the depthwise convolutional layer in the current depthwise separable convolutional network layer is calculated first. After the depthwise convolutional layer is calculated, the pointwise convolutional layer is calculated next. Therefore, when calculating the current depthwise separable convolutional network layer, the data flow convolution module can be configured to perform a depthwise convolution operation on the first input feature map in the depthwise convolution data flow convolution mode. Here, c is the number of parallel computations of the acceleration system in the input channel direction of the pointwise convolution. For example, when X = 128 and c = 8, the total number of rounds of pointwise convolution operations is 16. Here, since each depthwise separable convolutional network layer consists of a depthwise convolutional layer and a cascaded pointwise convolutional layer, the input feature map of the pointwise convolutional layer is the output feature map of the previous depthwise convolutional layer. To distinguish it from the first input feature map described above, the input feature map of the pointwise convolutional layer is referred to as the second input feature map.

[0041] The configurable addition tree module is used to accumulate the depthwise convolution results in an input addition mode that matches the depthwise convolution to obtain the depthwise convolution accumulation result; and to accumulate the pointwise convolution results of each round in an input addition mode that matches the pointwise convolution. Here, after the configurable addition tree module accumulates the depthwise convolution results, the accumulated result obtained is the accumulated result on the two-dimensional plane of the depthwise convolution. After the configurable data flow convolution module performs pointwise convolution operations on the data of the c channels in the second input feature map in each round, the convolution results of the c channels of this round of pointwise convolution are obtained. Therefore, after the configurable addition tree module accumulates the pointwise convolution results of one round, the accumulated result of the convolution results of the c channels of this round of pointwise convolution is obtained.

[0042] The intermediate result accumulation module is used to accumulate the point-by-point convolution results accumulated in each round with the point-by-point convolution results accumulated in the previous round until the point-by-point convolution results accumulated in all rounds are accumulated. Here, after accumulating the point-by-point convolution results accumulated in all rounds, the cumulative result of the point-by-point convolution results accumulated in all rounds is obtained. The cumulative result of the point-by-point convolution results accumulated in all rounds is the cumulative result of the convolution results of all input channels of the point-by-point convolution.

[0043] The accumulation result processing module is used to process the depthwise convolution accumulation results to obtain the output feature map of the depthwise convolution layer; it also processes the accumulation results of all rounds of pointwise convolution to obtain the output feature map of the current depthwise separable convolution network layer. Here, since each depthwise separable convolution network layer consists of a depthwise convolution layer and a cascaded pointwise convolution layer, the output feature map of the pointwise convolution layer in the current depthwise separable convolution network layer is the output feature map of the current depthwise separable convolution network layer.

[0044] Exemplarily, the weight storage buffer can be connected to the off-chip storage, and can be specifically used to obtain the weight parameters from the off-chip storage and send them to the configurable data flow convolution module; the feature map storage buffer can be connected to the off-chip storage, and can be specifically used to obtain the input feature map of the current depth-separable convolutional network layer from the off-chip storage and send it to the configurable data flow convolution module.

[0045] Exemplarily, the system further includes a point-by-point convolution intermediate result storage buffer. The point-by-point convolution intermediate result storage buffer is used to store the cumulative result of each round of point-by-point convolution and the cumulative result of the previous round as an accumulated intermediate result, and to send the latest accumulated intermediate result currently stored to the intermediate result accumulation module each time the intermediate result accumulation module performs an accumulation operation. After the intermediate result accumulation module completes one accumulation operation, the updated accumulated intermediate result is received from the intermediate result accumulation module for storage. For example, when the total number of rounds of point-by-point convolution operation is 3, the configurable addition tree module accumulates the point-by-point convolution results of the first round to obtain the accumulated point-by-point convolution results of the first round, and sends the accumulated point-by-point convolution results of the first round to the intermediate result accumulation unit. The intermediate result accumulation unit accumulates the accumulated point-by-point convolution results of the first round with the accumulated point-by-point convolution results of the 0th round (the accumulated point-by-point convolution results of the 0th round are empty) to obtain the first accumulated intermediate result, and stores the first accumulated intermediate result in the point-by-point convolution intermediate result storage buffer; thereafter, when the configurable addition tree module accumulates the point-by-point convolution results of the second round to obtain the accumulated point-by-point convolution results of the second round, and sends the accumulated point-by-point convolution results of the second round to the intermediate result accumulation unit, the intermediate result accumulation unit accumulates the accumulated point-by-point convolution results of the first round from the point-by-point convolution intermediate result storage buffer. The first accumulated intermediate result is obtained from the buffer, and the first accumulated intermediate result is accumulated with the second round of point-by-point convolution result to obtain the second accumulated intermediate result, and the second accumulated intermediate result is stored in the point-by-point convolution intermediate result storage buffer; thereafter, when the configurable addition tree module accumulates the third round of point-by-point convolution results to obtain the third round of accumulated point-by-point convolution result, and sends the third round of accumulated point-by-point convolution result to the intermediate result accumulation unit, the intermediate result accumulation unit obtains the second accumulated intermediate result from the point-by-point convolution intermediate result storage buffer, and accumulates the second accumulated intermediate result with the third round of accumulated point-by-point convolution result to obtain the third accumulated intermediate result, and then sends the third accumulated intermediate result to the accumulated result processing module for processing.

[0046] Here, for depth convolution, there is no need to accumulate the input channels, so the depth convolution accumulation result output by the configurable addition tree module does not need to be processed in the intermediate result accumulation module. The intermediate result accumulation module directly forwards the depth convolution accumulation result to the accumulation result processing module for processing.

[0047] For example, Figure 1 The figure is a schematic diagram of a configurable depth-separable convolution hardware acceleration system. Figure 1 As shown, the accumulation result processing module can include, in sequence: a data processing unit, a normalization unit NORM, an activation unit RELU, and a pooling unit POOLING. The data processing unit is used to quantize the data sent from the intermediate result accumulation module to reduce the data bit width, thereby reducing the required computing resources. For example, the accumulated 32-bit data can be quantized to 8 bits. The normalization unit NORM normalizes the quantized data, the activation unit RELU activates the normalized data, and the pooling unit POOLING pools the activated data to obtain the output feature map.

[0048] In some embodiments, when the current depthwise separable convolutional network layer is the first layer of the neural network processed by the system, the feature map storage buffer is specifically used to obtain the input feature map from the off-chip storage and send it to the configurable data flow convolution module; receive the output feature map of the depthwise convolution layer sent by the cumulative result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depthwise convolution layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation; and, store the output feature map of the current depthwise separable convolutional network layer sent by the cumulative result processing module, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depthwise separable convolutional network layer as the input feature map of the next layer.

[0049] Here, the neural network can be a neural network with a depthwise separable convolutional network layer as the first layer and other network layers as the remaining layers, or a combination of other network layers and one or more depthwise separable convolutional network layers as the remaining layers, or a neural network with one or more depthwise separable convolutional network layers as the remaining layers. The present invention is not limited to the specific neural network.

[0050] For example, when the neural network processed by the system is composed of three cascaded depthwise separable convolutional network layers, and the current depthwise separable convolutional network layer is the first layer of the neural network, and the output feature map of the current depthwise separable convolutional network layer has been obtained and stored in the feature map storage buffer, when the system processes the second layer of the neural network and calculates the next depthwise separable convolutional network layer, the output feature map of the first layer of the neural network is obtained from the feature map storage buffer and sent to the configurable data flow convolution module, and the weight parameters are continued to be obtained from the weight storage buffer and sent to the configurable data flow convolution module. Afterwards, the output feature map of the second layer of the neural network is calculated using the configurable data flow convolution module and the configurable addition tree module, the intermediate result accumulation module and the accumulated result processing module, and so on, until the output feature map of the third layer of the neural network is obtained and stored in the feature map storage buffer. Afterwards, the data stored in the feature map storage buffer can be transmitted to the off-chip for complete overall calculation.

[0051] In some embodiments, when the current depthwise separable convolutional network layer is the Zth layer of the neural network processed by the system, the feature map storage buffer is specifically used to use the output feature map of the Z-1th layer stored in itself as the input feature map, and send it to the configurable data flow convolution module; receive the output feature map of the depthwise convolution layer sent by the accumulation result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depthwise convolution layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depthwise separable convolutional network layer as the input feature map of the next layer, where Z is a positive integer greater than 1.

[0052] Here, the neural network can be a neural network including one or more depth-wise separable convolutional network layers, and the present invention is not limited to the specific neural network.

[0053] For example, when the first layer of the processed neural network is a depth-wise separable convolutional network, Figure 1 The specific operation of the configurable depth-separable convolution hardware acceleration system can be described as follows:

[0054] The input feature map data of the first layer is transferred from the off-chip storage to the feature map storage buffer, and then from the feature map storage buffer to the configurable data flow convolution module. The weight parameters are transferred from the off-chip storage to the weight storage buffer, and then from the weight storage buffer to the configurable data flow convolution module. After the feature map data and weight parameters are transmitted, the configurable data flow convolution module is configured accordingly according to the convolution type (depth convolution / point-by-point convolution) and the size information of the input feature map, and then performs the corresponding convolution operation and outputs the result. The result is then output to the configurable addition tree module. The configurable addition tree module completes the corresponding configuration according to the accumulation requirements that need to be completed, accumulates the convolution results, and then outputs the results to the intermediate result accumulation module for processing of the corresponding convolution type. For depth convolution, the input channels do not need to be accumulated, so no processing is required in this module, and the results can be directly output to the data processing unit. For point-by-point convolution, the accumulation result of the number of parallel channels on the input channel is only a partial sum of the input channels, which cannot be used as the final output of the point-by-point convolution. It is also necessary to The partial sum of the parallel channels of the input channels of the loop is accumulated until the results of all input channels are accumulated, and then the result is output to the data processing unit. Otherwise, the result is output to the point-by-point convolution intermediate result storage buffer for temporary storage as the input for the next round of intermediate result accumulation. After receiving the output of the intermediate result accumulation unit, the data processing unit quantizes the accumulated data and transmits the calculation result to the NORM, RELU, and POOLING units to complete data normalization, activation, and pooling operations. The pooling result is then transmitted as output to the feature map storage buffer as the input for the next layer of convolution calculation. At the same time, the weight parameters continue to be loaded from the off-chip storage. After the result calculation of the last layer of the neural network is completed, the feature map storage buffer transmits the result to the off-chip storage to complete the overall calculation. In short, the main function of the feature map storage buffer is to store the feature map results of the intermediate layer of the neural network and receive the first-layer feature map data transmitted from the off-chip storage, so as to avoid frequent on-chip and off-chip data interactions that generate a lot of power consumption.

[0055] In the present invention, a data flow convolution module can be configured, which is specifically used to work in a data flow convolution mode of depth convolution under the control of a first control signal, and perform a depth convolution operation on the weight parameters and the first input feature map in the data flow convolution mode of depth convolution to obtain a depth convolution result; and, under the control of a second control signal, work in a data flow convolution mode of point-by-point convolution, and perform a point-by-point convolution operation on the weight parameters of the 9 output channels with the data of c channels in the second input feature map in each round in the data flow convolution mode of point-by-point convolution to obtain the point-by-point convolution result of this round.

[0056] Specifically, each configurable data flow convolution module comprises a shift register array, a register array, a multiplier array and two multiplex data selectors.

[0057] The shift register array has three rows, the first and second rows have M shift registers, and the third row has 3 shift registers, the outputs of every two adjacent shift registers in each row are connected in sequence with the inputs; M is a preset positive integer greater than 3; each shift register is used for storing part of feature map data. Here, the value of M can be set according to actual needs, for example, it can be 224, etc., which is not limited in the present application.

[0058] The register array has three rows and three columns, each row and each column has three registers, and each register is used for storing part of weight parameters.

[0059] The multiplier array has three rows and three columns, each row and each column has three multipliers.

[0060] The output of the jth shift register in the ith row and the output of the jth register in the ith row are both connected with the two inputs of the jth multiplier in the ith row, wherein i and j are both positive integers, and the values of i and j are both 1 to 3, and the third input of each multiplier is also used for accessing an externally input switch signal, one switch signal is used for controlling whether the corresponding multiplier is in working state; the output of the first multiplex data selector is connected with the input of the first shift register in the second row, and the output of the second multiplex data selector is connected with the input of the first shift register in the third row; the output of the third shift register in the first row of the shift register array is connected with the first to P+1th inputs of the first multiplex data selector one by one, and the output of the third shift register in the second row of the shift register array is connected with the first to P+1th inputs of the second multiplex data selector one by one, wherein a is an integer, and the value of a is 0 to P, P is a preset positive integer greater than or equal to 1, the output of the third shift register in the first row of the shift register array is connected with the P+2th input of the first multiplex data selector, and the output of the third shift register in the second row of the shift register array is connected with the P+2th input of the second multiplex data selector; the control ends of the first and second multiplex data selectors are both used for accessing the same control signal, and the control signal is used for controlling the selected input of the multiplex data selector, so as to make the configurable data flow convolution module work in the data flow convolution mode of deep convolution or the data flow convolution mode of point-by-point convolution.

[0061] ​​For example, when M is 224 and P is 1, the outputs of the 224th and 112th registers in the first row of the shift register array are connected one-to-one to the first and second inputs of the first multiple-choice data selector, and the output of the third shift register in the first row of the shift register array is connected to the third input of the first multiple-choice data selector; similarly, the outputs of the 224th and 112th registers in the second row of the shift register array are connected one-to-one to the first and second inputs of the second multiple-choice data selector, and the output of the third shift register in the second row of the shift register array is connected to the third input of the second multiple-choice data selector. By taking different values ​​of M and P, the configurable data flow convolution module of the present invention can adapt to different input feature map sizes.

[0062] Specifically, when the control signal is the first control signal, the 1st to Yth input terminals of the first multi-select data selector and the second multi-select data selector are sequentially selected, and the configurable data flow convolution module enters a data flow convolution mode of depthwise convolution, where Y is less than or equal to P+1. When the control signal is the second control signal, the P+2th input terminals of the first multi-select data selector and the second multi-select data selector are selected, and the configurable data flow convolution module enters a data flow convolution mode of point-by-point convolution. Here, the input terminals selected by the first control signal and the second control signal and the number of selected input terminals can be set according to the size of the input feature map and the convolution type, and the present invention is not limited to this.

[0063] For example, Figure 2 A schematic diagram of the structure of a configurable data flow convolution module. Figure 2 A blue rectangle is a shift register for storing part of the feature map data, a green dot is a register for storing the corresponding weight parameters, a vertically placed yellow trapezoid is a multiple-choice data selector, and a Represents a multiplier, EN represents a switch signal, and the black arrows represent the data shift direction between adjacent shift registers. For depthwise convolution, a large amount of input feature map data will be reused during convolution calculation, for example Figure 3As shown, when the convolution window switches from rows 1, 2, and 3 to rows 2, 3, and 4 for convolution operation, the data of rows 2 and 3 need to be multiplexed, that is, the purple convolution window will reuse the feature map data "2", "1", "0", "9", "8", and "0" in the red convolution window. In order to avoid a large amount of power consumption caused by repeated transmission of the same data, it is crucial to multiplex the data when designing how the data flows. The configurable data flow convolution module in the present invention transfers the feature map data stored in the shift registers in rows 1 and 2 of the shift register array to the shift registers in rows 2 and 3 of the shift register array, respectively, to complete the multiplexing of the feature map data, and two multi-select data selectors select the data to be received from the shift registers in rows 1 and 2 of the shift register array according to the input feature map size, as shown in FIG. Figure 4 、 Figure 5 As shown, this is used to match feature maps of different input sizes and reuse them. Specifically, Figure 4 This is a schematic diagram of the connection between the selected input terminals of two multiple-choice data selectors and the output terminals of the corresponding shift registers when the configurable data flow convolution module performs a depthwise convolution operation under a certain input feature map size. Figure 5 This is a schematic diagram of the connection between the selected input terminals of two multiple-choice data selectors and the selected input terminals and the output terminals of the corresponding shift registers when the configurable data flow convolution module performs depthwise convolution operation under another input feature map size. Figure 2 When a rectangle corresponding to the position of 3×3 green dots is generated, nine multipliers calculate the corresponding results and then pass the data to the subsequent configurable adder tree module for accumulation to complete the depth convolution operation. For point-by-point convolution, since its convolution window is 1*1, there is no need to reuse the feature map data. For the output channel, 9 output parallel calculations are used. Therefore, a configurable data flow convolution module can store nine different weight parameters. The feature map data can flow through nine shift registers in turn and be multiplied with the corresponding weight parameters. For example, Figure 6 As shown, specifically, Figure 6 The figure below shows a connection diagram of the selected input terminals of two multi-select data selectors and the output terminals of the corresponding shift registers when the configurable data flow convolution module performs a point-by-point convolution operation. In this way, nine outputs can be obtained. The data is then passed to the subsequent configurable addition tree module for accumulation to complete the point-by-point convolution operation.

[0064] For depthwise separable convolution, since two different types of convolution operations, depthwise convolution and pointwise convolution, need to be performed, two dedicated data flow modules are often designed to adapt to the two types of convolution respectively. However, this inevitably increases the hardware resources and area overhead required for the design. The configurable data flow convolution module designed in the present invention can complete the convolution operations of pointwise convolution and depthwise convolution and their different feature map input sizes by adopting different configurations, and fully reuses the feature map data, so that the module can be reused to the greatest extent, saving hardware resources and reducing the design area. In addition, the computing unit (multiplier) can be turned off or on by an external input switch signal, which also reduces the power consumption generated by unnecessary calculations.

[0065] In depthwise separable convolution, point-by-point convolution and depthwise convolution have different ways of convolution data flow and their corresponding accumulation operations are also different. For depthwise convolution, the input channel is 1, so there is no accumulation in the channel direction. The convolution kernel size is generally 3*3, so the accumulated input in two-dimensional space is 9, such as Figure 7 As shown in , for a depthwise convolution with N outputs in parallel, one convolution operation obtains N cumulative results in two-dimensional space. For point-by-point convolution, the convolution kernel size is 1*1, so there is no accumulation in two-dimensional space. However, its input channel is often large, and it is necessary to perform accumulation operations in the direction of the input channel, such as Figure 8 As shown in the figure, for point-by-point convolution with N input channels in parallel (the parallelism of the point-by-point convolution input channels is equal to the parallelism of the depth-wise convolution output channels), one convolution operation obtains the accumulated results on 9 (the number of output channels is 9) N input channels.

[0066] For the different accumulation operations of depthwise convolution and pointwise convolution, the accumulated input of depthwise convolution is 9 inputs, and the accumulated input of pointwise convolution is equal to the parallel number N of input channels. In the design of the accelerator, the parallel number N of pointwise convolution is usually 8, 16, or 32. Therefore, the present invention designs a configurable addition tree module, which can be configured to complete the accumulation operations of 8, 9, 16, and 32 inputs, and can simultaneously adapt to the accumulation requirements of different parallel numbers of input channels for depthwise convolution and pointwise convolution.

[0067] For example, Figure 9 As shown, each configurable adder tree module includes: 4 8-input adder trees J1-J4, 5 2-input adders E1-E5 and 6 2-to-1 data selectors S1-S6. Figure 9 As shown, the connection relationship between these devices is as follows:

[0068] The control terminals of S1 to S6 are all used to access external input selection control signals, and the input terminals of J1 to J4 are all used to access 8 groups of input data. The output terminal of J1 is connected to one input terminal of E1, the first input terminal of S2 and the second input terminal of S1. The other input terminal of E1 is used to access 1 group of data. The output terminal of E1 is connected to the first input terminal of S1. The output terminal of S1 is the first output terminal O1 of the configurable addition tree module; the output terminal of S2 is connected to one input terminal of E2, the output terminal of J2 is connected to the other input terminal of E2 and the second input terminal of S3, the first input terminal of S3 is connected to one output terminal of E2, the output terminal of S3 is the second output terminal O2 of the configurable addition tree module, the other output terminal of E2 is connected to one input terminal of E3, and the other output terminal of E2 is the third output terminal O3 of the configurable addition tree module; the output terminal of J3 is connected to one input terminal of E4, the first input terminal of S4 The input ends are all connected, the other input end of E4 is connected to the output end of S5, one output end of E4 is connected to the second input end of S4, the other output end of E4 is connected to the other input end of E3, the output end of E3 is the fourth output end O4 of the configurable addition tree module, and the other output end of E4 is the fifth output end O5 of the configurable addition tree module. The output end of S4 is the sixth output end O6 of the configurable addition tree module; the output end of J4 is connected to the second input end of S5, one input end of E5 and the first input end of S6, the second input end of S5 is used to access a group of data, the other input end of E5 is used to access a group of data, the output end of E5 is connected to the second input end of S6, and the output end of S6 is the seventh output end O7 of the configurable addition tree module.

[0069] Here, the input addition mode of the configurable adder tree module can be configured through the external input selection control signal connected to the control terminals of S1 to S6 and the inputs of J1 to J4, E1, S2, S5, and E5.

[0070] Specifically, for depthwise convolution, Figure 9 and Figure 10 As shown by the blue line in , the left data selector selects 1 input data as output and adds it to the 8-input addition tree to complete the 9-input addition function. Then the right data selector selects the corresponding output result. A configurable addition tree module can output 4 9-input results at the same time. In other words, Figure 9 and Figure 10 As shown by the blue line in , when the first input of S1, the second input of S2, the first input of S3, the second input of S4, the first input of S5 and the second input of S6 are all selected, a configurable adder tree module works in 9-input addition mode, where O1, O2, O6 and O7 are all used to output a 9-input accumulation result. Therefore, for N-output parallel depth convolution, only A configurable addition tree module can complete all the accumulation operations to obtain the deep convolution accumulation result.

[0071] Specifically, for point-by-point convolution, when the input channel parallelism is 8, such as Figure 9 and Figure 11 As shown by the yellow line in , the result of the 8-input addition tree is directly transmitted to the data selector on the right for output. A configurable addition tree module can output 4 8-input results at the same time; that is, Figure 9 and Figure 11 As shown by the yellow line in , when the second input of S1, the second input of S3, the first input of S4 and the first input of S6 are all enabled, and both inputs of S2 and S5 are not enabled, the adder tree module can be configured to work in 8-input addition mode, where O1, O2, O6 and O7 are all used to output an 8-input accumulation result. When the input channel parallelism is 16, as shown in Figure 9 and Figure 12 As shown by the red line in , after the left data selector selects 8-input data as output, it is added to the output of the 8-input addition tree to complete the accumulation of 16 inputs, and then the result is directly output. A configurable addition tree module can output two 16-input results at the same time; that is, Figure 9 and Figure 12 As shown by the red line in , when the first input of S2 and the second input of S5 are both enabled, and the two inputs of S1, S3, S4 and S6 are not enabled, the configurable adder tree module can work in 16-input addition mode, where O3 and O5 are both used to output a 16-input accumulation result. When the input channel parallelism is 32, as shown in Figure 9 and Figure 13 As shown by the purple line in , the 32-input addition function can be completed by simply adding the two 16-input cumulative sums. A configurable addition tree module can output a 32-input result at the same time; that is, Figure 9 and Figure 13 As shown by the purple line in , when the first input of S2 and the second input of S5 are both enabled, and the two inputs of S1, S3, S4, and S6 are not enabled, the configurable adder tree module can also operate in 32-input addition mode, where O4 is used to output a 32-input accumulation result.

[0072] The configurable addition tree module designed in the present invention can be configured to complete the accumulation requirements of point-by-point convolution and depth-wise convolution, so that the two types of convolution can share the same addition tree unit. In addition, the input parallelism of the point-by-point convolution can also be flexibly configured to flexibly adapt to different accumulation needs, greatly improving the utilization rate of the module, thereby greatly reducing the resources and area overhead required for the design.

[0073] It should be noted that the number of configurable data flow convolution modules and configurable addition tree modules in the above system, as well as the number of other modules, can be set according to actual needs, and the present invention does not limit this.

[0074] The present invention also provides a configurable depth-separable convolution hardware acceleration method, which can be executed by the above-mentioned configurable depth-separable convolution hardware acceleration system, and the method includes.

[0075] S101. Obtain the first input feature map and weight parameters of the current depthwise separable convolutional network layer respectively; each depthwise separable convolutional network layer includes a depthwise convolutional layer and a cascaded point-by-point convolutional layer.

[0076] S102. Perform a depth convolution operation on the first input feature map in a depth convolution data flow convolution mode to obtain a depth convolution result.

[0077] S103: Accumulate the depth convolution results in an input addition mode that matches the depth convolution to obtain a depth convolution accumulation result.

[0078] S104: Process the depth convolution accumulation result to obtain the output feature map of the depth convolution layer, and cache the output feature map of the depth convolution layer.

[0079] S105. In the data flow convolution mode of point-by-point convolution, the output feature map of the depth convolution layer is used as the second input feature map. In each round, a point-by-point convolution operation is performed on the data of c channels in the output feature map of the cached second input depth convolution layer to obtain the point-by-point convolution result of this round.

[0080] S106 , accumulating the point-by-point convolution results of each round in an input addition mode that matches the point-by-point convolution.

[0081] S107: Accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round, until all accumulated point-by-point convolution results of all rounds are accumulated.

[0082] S108, processing the accumulation result of the point-by-point convolution results accumulated for all wheels to obtain an output feature map of the point-by-point convolution layer, the output feature map of the point-by-point convolution layer being an output feature map of the current depth separable convolution network layer, and caching the output feature map of the current depth separable convolution network layer as an input feature map for calculation of a next network layer.

[0083] It should be noted that the specific processing principles of each step can refer to the above description of the structure and function of each module in the system, and the present application will not be repeated here.

[0084] It should be noted that the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0085] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present application.

[0086] In the description, the word "comprising" does not exclude other components or steps, and "one" or "a" does not exclude a plurality. Some measures are described in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0087] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application cannot be limited to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or substitutions can be made, which should be considered as belonging to the protection scope of the present application.

Claims

1. A configurable depth-wise separable convolution hardware acceleration system, characterized in that: include: The feature map storage buffer and weight storage buffer are used to store the first input feature map and weight parameters of the current depth-separable convolutional network layer respectively; A configurable data flow convolution module is configured to perform a depthwise convolution operation on the first input feature map in a depthwise convolution data flow convolution mode to obtain a depthwise convolution result; in a pointwise convolution data flow convolution mode, perform a pointwise convolution operation on the data of c channels in the second input feature map of the pointwise convolution layer in each round to obtain the pointwise convolution result of this round; A configurable addition tree module is used to accumulate the depth convolution results in an input addition mode that matches the depth convolution to obtain the depth convolution accumulation result; and accumulate the results of each round of point-by-point convolution in an input addition mode that matches the point-by-point convolution; The intermediate result accumulation module is used to accumulate the point-by-point convolution results of each round with the point-by-point convolution results of the previous round until all the point-by-point convolution results of all rounds are accumulated; An accumulation result processing module is used to process the accumulation result of the depth convolution to obtain the output feature map of the depth convolution layer; and to process the accumulation result of the point-by-point convolution results accumulated from all rounds to obtain the output feature map of the current depth separable convolution network layer; The output feature map of the depth convolutional layer is the second input feature map; Each configurable data flow convolution module includes: a shift register array, a register array, a multiplier array and two multiple-to-one data selectors; The shift register array has three rows, the first and second rows have M shift registers, and the third row has 3 shift registers. The output ends and input ends of two adjacent shift registers in each row are connected in sequence; M is a preset positive integer greater than 3; each shift register is used to store part of the feature map data; The register array has three rows and three columns, each row and each column has three registers, and each register is used to store a part of the weight parameters; The multiplier array has three rows and three columns, and each row and each column has three multipliers; Among them, the output end of the j-th shift register in the i-th row and the output end of the j-th register in the i-th row are respectively connected to the two input ends of the j-th multiplier in the i-th row, wherein i and j are both positive integers, and the values ​​of i and j are both 1 to 3. The third input end of each multiplier is also used to access the external input switch signal, and a switch signal is used to control whether the corresponding multiplier is in a working state; the output end of the first multiple-choice data selector is connected to the input end of the first shift register in the second row, and the output end of the second multiple-choice data selector is connected to the input end of the first shift register in the third row; the first in the first row of the shift register array The output end of each register is connected to the 1st to P+1th input ends of the first multiple-select-one data selector one by one, and the first in the second row of the shift register array is connected to the 1st to P+1th input ends of the first multiple-select-one data selector one by one. The output end of each register is connected one by one to the 1st to P+1th input ends of the second multiple-choice data selector, wherein a is an integer, and the value of a ranges from 0 to P, and P is a preset positive integer greater than or equal to 1. The output end of the third shift register in the first row of the shift register array is connected to the P+2th input end of the first multiple-choice data selector, and the output end of the third shift register in the second row of the shift register array is connected to the P+2th input end of the second multiple-choice data selector; the control ends of the first multiple-choice data selector and the second multiple-choice data selector are both used to access the same control signal, and the control signal is used to control the multiple-choice data selector. The selected input terminal enables the configurable data flow convolution module to operate in a data flow convolution mode of depthwise convolution or a data flow convolution mode of pointwise convolution; when the control signal is a first control signal, the 1st to Yth input terminals of the first multiple-choice data selector and the second multiple-choice data selector are sequentially selected, and the configurable data flow convolution module enters the data flow convolution mode of depthwise convolution, wherein Y is less than or equal to P+1; when the control signal is a second control signal, the P+2th input terminal of the first multiple-choice data selector and the second multiple-choice data selector are selected, and the configurable data flow convolution module enters the data flow convolution mode of pointwise convolution; Each configurable adder tree module can be configured to perform 8-input, 9-input, 16-input, and 32-input accumulation operations. Each configurable adder tree module includes: four 8-input adder trees (J1) to (J4), five two-input adders (E1) to (E5), and six two-to-one data selectors (S1) to (S6); The control terminals (S1) to (S6) are used to receive external input selection control signals, and the input terminals (J1) to (J4) are used to receive 8 sets of input data. The output end of (J1) is connected to one input end of (E1), the first input end of (S2) and the second input end of (S1); the other input end of (E1) is used to access a set of data; the output end of (E1) is connected to the first input end of (S1); the output end of (S1) is the first output end (O1) of the configurable addition tree module; The output end of (S2) is connected to an input end of (E2), the output end of (J2) is connected to the other input end of (E2) and the second input end of (S3), the first input end of (S3) is connected to an output end of (E2), the output end of (S3) is the second output end (O2) of the configurable addition tree module, the other output end of (E2) is connected to an input end of (E3), and the other output end of (E2) is the third output end (O3) of the configurable addition tree module; The output end of (J3) is connected to one input end of (E4) and the first input end of (S4), the other input end of (E4) is connected to the output end of (S5), one output end of (E4) is connected to the second input end of (S4), the other output end of (E4) is connected to the other input end of (E3), the output end of (E3) is the fourth output end (O4) of the configurable addition tree module, and the other output end of (E4) is the fifth output end (O5) of the configurable addition tree module, and the output end of (S4) is the sixth output end (O6) of the configurable addition tree module; The output end of (J4) is connected to the second input end of (S5), one input end of (E5) and the first input end of (S6), the second input end of (S5) is used to access a group of data, the other input end of (E5) is used to access a group of data, the output end of (E5) is connected to the second input end of (S6), and the output end of (S6) is the seventh output end (O7) of the configurable addition tree module.

2. The configurable depthwise separable convolution hardware acceleration system according to claim 1, wherein: The system also includes: a point-by-point convolution intermediate result storage buffer, which is used to store the accumulated point-by-point convolution results of each round and the accumulated point-by-point convolution results of the previous round as accumulated intermediate results, and each time the intermediate result accumulation module performs an accumulation operation, the currently stored latest accumulated intermediate result is sent to the intermediate result accumulation module, and after the intermediate result accumulation module completes an accumulation operation, the updated accumulated intermediate result is received from the intermediate result accumulation module for storage.

3. The configurable depthwise separable convolution hardware acceleration system according to claim 1, wherein: The configurable data flow convolution module is specifically configured to operate in a data flow convolution mode of depthwise convolution under the control of a first control signal, and perform a depthwise convolution operation on the first input feature map according to a weight parameter in the data flow convolution mode of depthwise convolution to obtain a depthwise convolution result; and, under the control of a second control signal, operate in a data flow convolution mode of pointwise convolution, and perform a pointwise convolution operation on the data of c channels in the output feature map of the depthwise convolution layer according to the weight parameter in each round of the data flow convolution mode of pointwise convolution to obtain the pointwise convolution result of this round; wherein, when the output feature map of the depthwise convolution layer has X channels, the total number of rounds of pointwise convolution operation is .

4. The configurable depthwise separable convolution hardware acceleration system according to claim 1, wherein: The input addition modes of each configurable addition tree module include: 8-input addition mode, 9-input addition mode, 16-input addition mode and 32-input addition mode; In the 8-input addition mode, each configurable addition tree module is used to simultaneously output 4 8-input accumulation results; in the 9-input addition mode, each configurable addition tree module is used to simultaneously output 4 9-input accumulation results; in the 16-input addition mode, each configurable addition tree module is used to simultaneously output 2 16-input accumulation results; in the 32-input addition mode, each configurable addition tree module is used to simultaneously output 1 32-input accumulation result; wherein, the input addition mode matching the depthwise convolution is the 9-input addition mode, and the input addition mode matching the point-by-point convolution is any one of the 8-input addition mode, the 16-input addition mode and the 32-input addition mode.

5. The configurable depthwise separable convolution hardware acceleration system according to claim 1, wherein: When the second input terminal of S1, the second input terminal of S3, the first input terminal of S4 and the first input terminal of S6 are all enabled, and both input terminals of S2 and S5 are not enabled, the configurable adder tree module operates in an 8-input addition mode, wherein O1, O2, O6 and O7 are all used to output an 8-input accumulation result; When the first input terminal of S1, the second input terminal of S2, the first input terminal of S3, the second input terminal of S4, the first input terminal of S5 and the second input terminal of S6 are all enabled, the configurable adder tree module operates in a 9-input addition mode, wherein O1, O2, O6 and O7 are all used to output a 9-input accumulation result; When the first input terminal of S2 and the second input terminal of S5 are both selected, and the two input terminals of S1, S3, S4 and S6 are not selected, the configurable adder tree module operates in 16-input addition mode and 32-input addition mode, wherein O3 and O5 are both used to output a 16-input accumulation result, and O4 is used to output a 32-input accumulation result.

6. The configurable depthwise separable convolution hardware acceleration system according to claim 1, wherein: When the current depth-wise separable convolutional network layer is the first layer of the neural network processed by the system, the feature map storage buffer is specifically used to obtain the first input feature map from the off-chip storage and send it to the configurable data flow convolution module; receive the output feature map of the depth-wise convolutional layer sent by the cumulative result processing block and store it; in each round, send the data of c channels in the stored output feature map of the depth-wise convolutional layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation; and store the output feature map of the current depth-wise separable convolutional network layer sent by the cumulative result processing module, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depth-wise separable convolutional network layer as the input feature map of the next layer; When the current depthwise separable convolutional network layer is the Zth layer of the neural network processed by the system, the feature map storage buffer is specifically used to use the output feature map of the Z-1th layer stored in itself as the first input feature map, and send it to the configurable data flow convolution module; receive the output feature map of the depthwise convolutional layer sent by the accumulation result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depthwise convolutional layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depthwise separable convolutional network layer as the input feature map of the next layer, where Z is a positive integer greater than 1.

7. A configurable depth-separable convolution hardware acceleration method, characterized in that: Applied to the configurable depthwise separable convolution hardware acceleration system according to any one of claims 1 to 6, the method comprising: Obtain the first input feature map and weight parameters of the current depthwise separable convolutional network layer respectively; each depthwise separable convolutional network layer includes a depthwise convolutional layer and a cascaded pointwise convolutional layer; Performing a depthwise convolution operation on the first input feature map in a depthwise convolution data flow convolution mode to obtain a depthwise convolution result; Accumulating the depth convolution results in an input addition mode matching the depth convolution to obtain a depth convolution accumulation result; Processing the depth convolution accumulation result to obtain an output feature map of the depth convolution layer, and caching the output feature map of the depth convolution layer; In the data flow convolution mode of point-by-point convolution, the output feature map of the depthwise convolution layer is used as the second input feature map. In each round, a point-by-point convolution operation is performed on the data of c channels in the cached second input feature map to obtain the point-by-point convolution result of this round. Accumulate the results of each round of point-by-point convolution in an input addition mode that matches the point-by-point convolution; Accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated; The accumulated results of all rounds of point-by-point convolution results are processed to obtain an output feature map of the point-by-point convolution layer, where the output feature map of the point-by-point convolution layer is the output feature map of the current depth-wise separable convolutional network layer, and the output feature map of the current depth-wise separable convolutional network layer is cached to serve as the input feature map for calculation of the next network layer.

Images