Configurable depth separable convolution hardware acceleration system and method

Abstract

The invention discloses a configurable depth separable convolution hardware acceleration system and method, and the system comprises a feature map storage buffer and a weight storage buffer which are used for storing the input and weight of a current convolution network layer; the configurable data flow convolution module is used for performing depth separable convolution operation on the input, and performing corresponding convolution operation on data of partial channels of the input feature map in each round; the configurable adder tree module is used for accumulating the convolution results and correspondingly accumulating the convolution results of each round; the intermediate result accumulation module is used for accumulating the point-by-point convolution result of the part of channels accumulated in each round and the point-by-point convolution result of the part of channels accumulated in the previous round until all the input channels are accumulated and the result is output; and the accumulation result processing module is used for processing the convolution accumulation result to obtain the output of the current convolution layer. According to the invention, the design area and the resource overhead are greatly reduced through the configurability of the main calculation module.

Description

Technical Field

[0001] The present invention belongs to the technical field of neural network hardware acceleration, and in particular relates to a configurable depth-separable convolution hardware acceleration system and method. Background Art

[0002] With the continuous advancement of neural network technology, it is becoming more and more common in many practical application fields, such as image recognition, speech processing, and autonomous driving. However, as the number of network layers increases, the number of model parameters also surges, which brings a series of challenges to the deployment of neural networks on mobile devices. The huge computing requirements lead to a significant increase in the latency of neural networks when performing inference, which is unacceptable in some application scenarios. Therefore, the neural networks deployed on mobile devices are often lightweight networks. There are a large number of depthwise separable convolution operations in these network models, but the current acceleration schemes for depthwise separable convolutions generally lack flexibility, and no universal acceleration strategy has been developed for pointwise convolution and depthwise convolution operations in depthwise separable convolutions. This leads to insufficient utilization of hardware modules when performing depthwise separable convolutions, thereby unnecessarily increasing the consumption of hardware resources.

[0003] Existing solutions usually adopt different acceleration strategies for different types of convolution operations in depthwise separable convolution. Special modules are often designed to handle the feature map data flow of pointwise convolution and depthwise convolution and convolution multiplication calculation. The addition tree module after convolution multiplication calculation is usually configured with a 9-input addition tree and an addition tree with the same parallel number as the input channel, which respectively handle the accumulation of depthwise convolution in the two-dimensional plane and the accumulation of pointwise convolution in the direction of the input channel during the depthwise separable convolution operation. In short, the acceleration solutions of traditional solutions are often too single and can only be used for specific networks and FPGA devices. The design portability is poor. In addition, since the module parameters are not configurable, more modules need to be designed to meet the computing needs, resulting in poor module reusability and increased resource and area overhead. Summary of the invention

[0004] In order to solve the above problems existing in the prior art, the present invention provides a configurable depth-separable convolution hardware acceleration system and method.

[0005] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a configurable depth-separable convolution hardware acceleration system, comprising:

[0007] A feature map storage buffer and a weight storage buffer, used to store the first input feature map and weight parameters of the current depth-separable convolutional network layer respectively;

[0008] A configurable data flow convolution module is used to perform a depth convolution operation on the first input feature map in a depth convolution data flow convolution mode to obtain a depth convolution result; in a point-by-point convolution data flow convolution mode, perform a point-by-point convolution operation on the data of c channels in the second input feature map of the point-by-point convolution layer in each round to obtain the point-by-point convolution result of this round;

[0009] A configurable addition tree module is used to accumulate the depth convolution results in an input addition mode matching the depth convolution to obtain the depth convolution accumulation result; and accumulate the point-by-point convolution results of each round in an input addition mode matching the point-by-point convolution;

[0010] An intermediate result accumulation module is used to accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated;

[0011] The accumulation result processing module is used to process the accumulation result of the deep convolution to obtain the output feature map of the deep convolution layer; process the accumulation result of all rounds of accumulated point-by-point convolution results to obtain the output feature map of the current deep separable convolution network layer; the output feature map of the deep convolution layer is the second input feature map.

[0012] The present invention also provides a configurable depth-separable convolution hardware acceleration method, which is applied to the above-mentioned configurable depth-separable convolution hardware acceleration system, and the method comprises:

[0013] The first input feature map and weight parameters of the current depth-separable convolutional network layer are obtained respectively; each depth-separable convolutional network layer includes a depth-wise convolutional layer and a cascaded point-by-point convolutional layer;

[0014] Performing a deep convolution operation on the first input feature map in a data flow convolution mode of deep convolution to obtain a deep convolution result;

[0015] Accumulating the depth convolution results in an input addition mode matching the depth convolution to obtain a depth convolution accumulation result;

[0016] Processing the deep convolution accumulation result to obtain an output feature map of the deep convolution layer, and caching the output feature map of the deep convolution layer;

[0017] In the data flow convolution mode of point-by-point convolution, the output feature map of the depth convolution layer is used as the second input feature map, and the data of c channels in the cached second input feature map are subjected to point-by-point convolution operation in each round to obtain the point-by-point convolution result of this round;

[0018] Accumulate the results of each round of point-by-point convolution in an input addition mode that matches the point-by-point convolution;

[0019] Accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated;

[0020] The accumulated results of all rounds of accumulated point-by-point convolution results are processed to obtain an output feature map of the point-by-point convolution layer, where the output feature map of the point-by-point convolution layer is the output feature map of the current depth-separable convolutional network layer, and the output feature map of the current depth-separable convolutional network layer is cached to serve as an input feature map for calculation of the next network layer.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] Aiming at the large number of multiplication and accumulation calculation requirements during neural network reasoning, the present invention first optimizes the convolution calculation data flow convolution module so that it can adopt different configurations to simultaneously complete point-by-point convolution and depth-wise convolution operations to achieve the effect of maximizing the reuse of convolution modules. Similarly, a configurable addition tree module is designed so that it can simultaneously support the accumulation operations of point-by-point convolution and depth-wise convolution, and can adapt to different input channel parallelism designs, making the design very flexible and the module utilization rate high, greatly reducing the design area and resource overhead, and the computing unit can be turned off to reduce the power consumption generated by unnecessary calculations.

[0023] The present invention will be further described in detail below with reference to the accompanying drawings and specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is an exemplary structural diagram of a configurable depth-separable convolution hardware acceleration system provided by an embodiment of the present invention;

[0025] Figure 2 is a schematic diagram of the structure of a configurable data flow convolution module provided by an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the principle of deep convolution for data reuse;

[0027] Figure 4 is a schematic diagram of an exemplary internal configuration of a configurable data flow convolution module provided by an embodiment of the present invention when working in a data flow convolution mode of depth convolution;

[0028] Figure 5 is a schematic diagram of another exemplary internal configuration of a configurable data flow convolution module provided by an embodiment of the present invention when working in a data flow convolution mode of depth convolution;

[0029] Figure 6 It is a schematic diagram of the internal configuration of a configurable data flow convolution module provided by an embodiment of the present invention when working in a data flow convolution mode of point-by-point convolution;

[0030] Figure 7 It is an exemplary schematic diagram of the accumulation principle of deep convolution;

[0031] Figure 8 It is an exemplary schematic diagram of the accumulation principle of point-by-point convolution;

[0032] Fig. 9 It is a structural diagram of a configurable addition tree module provided by an embodiment of the present invention;

[0033] Fig.10 It is a schematic diagram of the internal configuration of a configurable addition tree module provided by an embodiment of the present invention when working in a 9-input addition mode;

[0034] Fig.11 It is a schematic diagram of the internal configuration of a configurable addition tree module provided by an embodiment of the present invention when working in an 8-input addition mode;

[0035] Fig.12 It is a schematic diagram of the internal configuration of a configurable addition tree module provided by an embodiment of the present invention when working in a 16-input addition mode;

[0036] Fig.13 It is a schematic diagram of the internal configuration of a configurable addition tree module provided by an embodiment of the present invention when working in a 32-input addition mode. DETAILED DESCRIPTION

[0037] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0038] The present invention provides a configurable depth-separable convolution hardware acceleration system, and the system is a depth-separable convolution hardware acceleration system based on FPGA. The system includes: a feature map storage buffer, a weight storage buffer, a configurable data flow convolution module and a configurable addition tree module, an intermediate result accumulation module and an accumulation result processing module. The feature map storage buffer and the weight storage buffer are both connected to the configurable data flow convolution module to send feature map data and weight parameters to the configurable data flow convolution module. The configurable data flow convolution module is connected to the configurable addition tree module to send the convolution calculation result to the configurable addition tree module for addition operation. The configurable addition tree module is connected to the intermediate result accumulation module to send the data after addition calculation to the intermediate result accumulation module. The intermediate result accumulation module is also connected to the accumulation result processing module to send the calculated accumulation result to the accumulation result processing module for processing. The accumulation result processing module is also connected to the feature map storage buffer to store the feature map obtained after processing in the feature map storage buffer.

[0039] The feature map storage buffer and the weight storage buffer are used to obtain and store the input feature map and weight parameters of the current depth-separable convolutional network layer respectively; each depth-separable convolutional network layer includes a depth-wise convolutional layer and a cascaded point-by-point convolutional layer. Here, the input feature map of the current depth-separable convolutional network layer is referred to as the first input feature map.

[0040] The configurable data flow convolution module is used to perform a depthwise convolution operation on the first input feature map in the data flow convolution mode of depthwise convolution to obtain a depthwise convolution result; in the data flow convolution mode of pointwise convolution, a pointwise convolution operation is performed on the data of c channels in the input feature map of the pointwise convolution layer in each round to obtain the pointwise convolution result of this round; when the output feature map of the depthwise convolution layer has X channels, the total number of rounds of pointwise convolution operation is Here, since each depth-separable convolutional network layer is composed of a depth-wise convolutional layer and a cascaded point-by-point convolutional layer, when calculating the current depth-wise separable convolutional network layer, the depth-wise convolutional layer in the current depth-wise separable convolutional network layer will be calculated first, and after the depth-wise convolutional layer is calculated, the point-by-point convolutional layer will be calculated next. Therefore, when calculating the current depth-wise separable convolutional network layer, the data flow convolution module can be configured to perform a depth-wise convolution operation on the first input feature map in the data flow convolution mode of the depth-wise convolution. Here, c is the number of parallel calculations of the acceleration system in the direction of the point-by-point convolution input channel. For example, when X=128 and c=8, the total number of rounds of the point-by-point convolution operation is 16. Here, since each depth-wise separable convolutional network layer is composed of a depth-wise convolutional layer and a cascaded point-by-point convolutional layer, the input feature map of the point-by-point convolutional layer is the output feature map of the previous depth-wise convolutional layer. In order to distinguish it from the first input feature map mentioned above, the input feature map of the point-by-point convolutional layer is referred to as the second input feature map below.

[0041] The configurable addition tree module is used to accumulate the depth convolution results in the input addition mode that matches the depth convolution to obtain the depth convolution accumulation result; and to accumulate the point-by-point convolution results of each round in the input addition mode that matches the point-by-point convolution. Here, after the configurable addition tree module accumulates the depth convolution results, the accumulated result obtained is the accumulated result of the depth convolution on the two-dimensional plane. After the configurable data flow convolution module performs point-by-point convolution operation on the data of the c channels in the second input feature map in each round, the convolution result of the c channels of this round of point-by-point convolution is obtained. Therefore, after the configurable addition tree module accumulates the point-by-point convolution results of one round, the accumulated result of the convolution results of the c channels of this round of point-by-point convolution is obtained.

[0042] The intermediate result accumulation module is used to accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated. Here, after accumulating the accumulated point-by-point convolution results of all rounds, the accumulated result of the accumulated point-by-point convolution results of all rounds is the accumulated result of the convolution results of all input channels of the point-by-point convolution.

[0043] The accumulation result processing module is used to process the accumulation results of the deep convolution to obtain the output feature map of the deep convolution layer; the accumulation results of all rounds of accumulated point-by-point convolution results are processed to obtain the output feature map of the current deep separable convolution network layer. Here, since each deep separable convolution network layer is composed of a deep convolution layer and a cascaded point-by-point convolution layer, the output feature map of the point-by-point convolution layer in the current deep separable convolution network layer is the output feature map of the current deep separable convolution network layer.

[0044] Exemplarily, the weight storage buffer can be connected to the off-chip storage, and can be specifically used to obtain the weight parameters from the off-chip storage and send them to the configurable data flow convolution module; the feature map storage buffer can be connected to the off-chip storage, and can be specifically used to obtain the input feature map of the current depth-separable convolutional network layer from the off-chip storage and send it to the configurable data flow convolution module.

[0045] Exemplarily, the system further includes: a point-by-point convolution intermediate result storage buffer. The point-by-point convolution intermediate result storage buffer is used to store the accumulated point-by-point convolution result of each round and the accumulated point-by-point convolution result of the previous round as an accumulated intermediate result, and when the intermediate result accumulation module performs an accumulation operation each time, the latest accumulated intermediate result currently stored is sent to the intermediate result accumulation module, and after the intermediate result accumulation module performs an accumulation operation, the updated accumulated intermediate result is received from the intermediate result accumulation module for storage. For example, when the total number of rounds of the point-by-point convolution operation is 3, after the configurable addition tree module accumulates the point-by-point convolution results of the first round to obtain the accumulated point-by-point convolution results of the first round, and sends the accumulated point-by-point convolution results of the first round to the intermediate result accumulation unit, the intermediate result accumulation unit will accumulate the accumulated point-by-point convolution results of the first round with the accumulated point-by-point convolution results of the 0th round (the accumulated point-by-point convolution results of the 0th round are empty) to obtain the first accumulated intermediate result, and store the first accumulated intermediate result in the point-by-point convolution intermediate result storage buffer; thereafter, when the configurable addition tree module accumulates the point-by-point convolution results of the second round to obtain the accumulated point-by-point convolution results of the second round, and sends the accumulated point-by-point convolution results of the second round to the intermediate result accumulation unit, the intermediate result accumulation unit will store the accumulated point-by-point convolution results from the point-by-point convolution intermediate result storage buffer. The first accumulated intermediate result is obtained from the buffer, and the first accumulated intermediate result is accumulated with the second round of accumulated point-by-point convolution result to obtain the second accumulated intermediate result, and the second accumulated intermediate result is stored in the point-by-point convolution intermediate result storage buffer; thereafter, when the configurable addition tree module accumulates the third round of point-by-point convolution results to obtain the third round of accumulated point-by-point convolution results, and sends the third round of accumulated point-by-point convolution results to the intermediate result accumulation unit, the intermediate result accumulation unit obtains the second accumulated intermediate result from the point-by-point convolution intermediate result storage buffer, and accumulates the second accumulated intermediate result with the third round of accumulated point-by-point convolution results to obtain the third accumulated intermediate result, and then sends the third accumulated intermediate result to the accumulated result processing module for processing.

[0046] Here, for deep convolution, there is no need to accumulate the input channels, so the deep convolution accumulation result output by the configurable addition tree module does not need to be processed in the intermediate result accumulation module. The intermediate result accumulation module directly forwards the deep convolution accumulation result to the accumulation result processing module for processing.

[0047] For example, Figure 1 FIG. 1 is a schematic diagram of a configurable depth-separable convolution hardware acceleration system. Figure 1 As shown, the accumulation result processing module may include: a data processing unit, a normalization unit NORM, an activation unit RELU, and a pooling unit POOLING in sequence. The data processing unit is used to quantize the data sent from the intermediate result accumulation module to reduce the data bit width, thereby reducing the resources required for calculation. For example, the accumulated 32-bit data may be quantized to 8 bits. The normalization unit NORM normalizes the quantized data, the activation unit RELU activates the normalized data, and the pooling unit POOLING pools the activated data to obtain the output feature map.

[0048] In some embodiments, when the current depthwise separable convolutional network layer is the first layer of the neural network processed by the system, the feature map storage buffer is specifically used to obtain the input feature map from the off-chip storage and send it to the configurable data flow convolution module; receive the output feature map of the depth convolution layer sent by the accumulation result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depth convolution layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation; and, store the output feature map of the current depthwise separable convolutional network layer sent by the accumulation result processing module, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depthwise separable convolutional network layer as the input feature map of the next layer.

[0049] Here, the neural network can be a neural network with a depthwise separable convolutional network layer as the first layer and other network layers as the remaining layers, or a combination of other network layers and one or more depthwise separable convolutional network layers as the remaining layers, or a neural network with one or more depthwise separable convolutional network layers as the remaining layers. The present invention is not limited to the specific neural network.

[0050] For example, when the neural network processed by the system is composed of three cascaded depth-wise separable convolutional network layers, and the current depth-wise separable convolutional network layer is the first layer of the neural network, and the output feature map of the current depth-wise separable convolutional network layer has been obtained and stored in the feature map storage buffer, when the system processes the second layer of the neural network, the calculation of the next depth-wise separable convolutional network layer, the output feature map of the first layer of the neural network is obtained from the feature map storage buffer and sent to the configurable data flow convolution module, and the weight parameters are continued to be obtained from the weight storage buffer and sent to the configurable data flow convolution module. After that, the output feature map of the second layer of the neural network is calculated using the configurable data flow convolution module and the configurable addition tree module, the intermediate result accumulation module and the accumulation result processing module, and so on, until the output feature map of the third layer of the neural network is obtained and stored in the feature map storage buffer, after which the data stored in the feature map storage buffer can be transmitted to the off-chip complete overall calculation.

[0051] In some embodiments, when the current depthwise separable convolutional network layer is the Zth layer of the neural network processed by the system, the feature map storage buffer is specifically used to use the output feature map of the Z-1th layer stored in itself as the input feature map, and send it to the configurable data flow convolution module; receive the output feature map of the depth convolution layer sent by the accumulation result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depth convolution layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depthwise separable convolutional network layer as the input feature map of the next layer, where Z is a positive integer greater than 1.

[0052] Here, the neural network may be a neural network including one or more depth-separable convolutional network layers, and the present invention is not limited to the specific neural network.

[0053] Exemplarily, in the case where the first layer of the processed neural network is a depthwise separable convolutional network, Figure 1 The specific operation of the configurable depth-separable convolution hardware acceleration system shown can be described as follows:

[0054] The input feature map data of the first layer is transferred from the off-chip storage to the feature map storage buffer, and then from the feature map storage buffer to the configurable data flow convolution module. The weight parameters are transferred from the off-chip storage to the weight storage buffer, and then from the weight storage buffer to the configurable data flow convolution module. After the feature map data and weight parameters are transmitted, the configurable data flow convolution module is configured accordingly according to the convolution type (depth convolution / point-by-point convolution) and the size information of the input feature map, and then the corresponding convolution operation is performed and the result is output, and then the result is output to the configurable addition tree module. The configurable addition tree module completes the corresponding configuration according to the accumulation requirements to be completed, accumulates the convolution results, and then outputs the results to the intermediate result accumulation module for the corresponding convolution type processing. For depth convolution, the input channels do not need to be accumulated, so no processing is required in this module, and the results can be directly output to the data processing unit. For point-by-point convolution, the accumulation result of the number of parallel channels on the input channel is only the partial sum of the input channels, which cannot be used as the final output of the point-by-point convolution. It is also necessary to output each time. The partial sum of the number of parallel channels of the input channels of the loop is accumulated until the results of all input channels are accumulated, and then the result is output to the data processing unit. Otherwise, the result is output to the point-by-point convolution intermediate result storage buffer for temporary storage as the input for the next round of intermediate result accumulation; after the data processing unit receives the output of the intermediate result accumulation unit, it quantizes the accumulated data, and then transmits the calculation result to the NORM, RELU, and POOLING units to complete data normalization, activation, and pooling operations, and then transmits the pooling result as output to the feature map storage buffer as the input for the next layer of convolution calculation. At the same time, the weight parameters continue to be loaded from the off-chip storage. After the result calculation of the last layer of the neural network is completed, the feature map storage buffer transfers the result to the off-chip storage to complete the overall calculation. In short, the main function of the feature map storage buffer is to store the feature map results of the intermediate layer of the neural network and receive the first layer feature map data transmitted from the off-chip storage, so as to avoid frequent on-chip and off-chip data interactions that generate a lot of power consumption.

[0055] In the present invention, a data flow convolution module can be configured, which is specifically used to work in a data flow convolution mode of depth convolution under the control of a first control signal, and perform a depth convolution operation on the weight parameters and the first input feature map in the data flow convolution mode of depth convolution to obtain a depth convolution result; and, under the control of a second control signal, work in a data flow convolution mode of point-by-point convolution, and perform a point-by-point convolution operation on the weight parameters of the 9 output channels with the data of c channels in the second input feature map in each round in the data flow convolution mode of point-by-point convolution to obtain the point-by-point convolution result of this round.

[0056] Specifically, each configurable data flow convolution module includes: a shift register array, a register array, a multiplier array and two one-of-many data selectors.

[0057] The shift register array has three rows, the first row and the second row have M shift registers, the third row has 3 shift registers, and the output ends and input ends of two adjacent shift registers in each row are connected in sequence; M is a preset positive integer greater than 3; each shift register is used to store part of the feature map data. Here, the value of M can be set according to actual needs, for example, it can be 224, etc., and the present invention is not limited to this.

[0058] The register array has three rows and three columns, each row and each column has three registers, and each register is used to store a part of the weight parameters.

[0059] The multiplier array has three rows and three columns, and each row and each column has three multipliers.

[0060] The output end of the j-th shift register in the i-th row and the output end of the j-th register in the i-th row are respectively connected to the two input ends of the j-th multiplier in the i-th row, wherein i and j are both positive integers, and the values ​​of i and j are both 1 to 3, and the third input end of each multiplier is also used to access an external input switch signal, and a switch signal is used to control whether a corresponding multiplier is in a working state; the output end of the first multiple-choice data selector is connected to the input end of the first shift register in the second row, and the output end of the second multiple-choice data selector is connected to the input end of the first shift register in the third row; the output end of the j-th shift register in the first row of the shift register array is connected to the input end of the j-th multiplier in the third row; The output end of each register is connected to the 1st to P+1th input ends of the first multiple-select-one data selector one by one. The output end of each register is connected one by one with the 1st to P+1th input ends of the second multiple-choice data selector, wherein a is an integer, and the value of a ranges from 0 to P, and P is a preset positive integer greater than or equal to 1. The output end of the third shift register in the first row of the shift register array is connected with the P+2th input end of the first multiple-choice data selector, and the output end of the third shift register in the second row of the shift register array is connected with the P+2th input end of the second multiple-choice data selector; the control ends of the first multiple-choice data selector and the second multiple-choice data selector are both used to access the same control signal, and the control signal is used to control the selected input end of the multiple-choice data selector so that the configurable data flow convolution module operates in the data flow convolution mode of depth convolution or the data flow convolution mode of point-by-point convolution.

[0061] For example, when M is 224 and P is 1, the output ends of the 224th and 112th registers in the first row of the shift register array are connected one by one to the first to second input ends of the first multiple-select-one data selector, and the output end of the third shift register in the first row of the shift register array is connected to the third input end of the first multiple-select-one data selector; similarly, the output ends of the 224th and 112th registers in the second row of the shift register array are connected one by one to the first to second input ends of the second multiple-select-one data selector, and the output end of the third shift register in the second row of the shift register array is connected to the third input end of the second multiple-select-one data selector. Through different values ​​of M and P, the configurable data flow convolution module of the present invention can adapt to different input feature map sizes.

[0062] Specifically, when the control signal is the first control signal, the 1st to Yth input terminals of the first multi-select data selector and the second multi-select data selector are sequentially selected, and the configurable data flow convolution module enters a data flow convolution mode of depthwise convolution, where Y is less than or equal to P+1. When the control signal is the second control signal, the P+2th input terminal of the first multi-select data selector and the second multi-select data selector are selected, and the configurable data flow convolution module enters a data flow convolution mode of point-by-point convolution. Here, the input terminals selected by the first control signal and the second control signal and the number of selected input terminals can be set according to the size of the input feature map and the convolution type, and the present invention is not limited to this.

[0063] For example, Figure 2 A schematic diagram of the structure of a configurable data flow convolution module. Figure 2 A blue rectangle is a shift register for storing part of the feature map data, a green dot is a register for storing the corresponding weight parameter, a vertically placed yellow trapezoid is a multiple-choice data selector, and a represents a multiplier, EN represents a switch signal, and the black arrows represent the data shift direction between adjacent shift registers. For deep convolution, a large amount of input feature map data is reused during convolution calculation, for example Figure 3As shown, when the convolution window switches from rows 1, 2, and 3 to rows 2, 3, and 4 for convolution operation, the data in rows 2 and 3 need to be multiplexed, that is, the purple convolution window will reuse the feature map data "2", "1", "0", "9", "8", and "0" in the red convolution window. In order to avoid a large amount of power consumption caused by repeated transmission of the same data, it is very important to reuse the data when designing how the data flows. The configurable data flow convolution module in the present invention transfers the feature map data stored in the shift registers in the 1st and 2nd rows of the shift register array to the shift registers in the 2nd and 3rd rows of the shift register array, respectively, so as to complete the multiplexing of the feature map data, and two multi-select data selectors select the data to be received from the shift registers in the 1st and 2nd rows of the shift register array according to the input feature map size, as shown in FIG. Figure 4 , Figure 5 As shown, in order to match the feature maps of different input sizes and reuse them, specifically, Figure 4 It is a schematic diagram of the connection between the selected input terminals of two multiple-choice data selectors and the selected input terminals and the output terminals of the corresponding shift registers when the configurable data flow convolution module performs a depth convolution operation under a certain input feature map size. Figure 5 It is a connection diagram of the selected input terminals of two multiple-choice data selectors and the selected input terminals and the output terminals of the corresponding shift registers when the configurable data flow convolution module performs a depthwise convolution operation under another input feature map size. When the weight parameter and the first feature map effective window ( Figure 2 When a rectangle corresponding to the position of 3×3 green dots is generated, nine multipliers calculate the corresponding results, and then pass the data to the subsequent configurable addition tree module for accumulation to complete the deep convolution operation. For point-by-point convolution, since its convolution window is 1*1, there is no need to reuse the feature map data. For the output channel, 9 output parallel calculations are used. Therefore, a configurable data flow convolution module can store nine different weight parameters. The feature map data can flow through nine shift registers in turn and multiply with the corresponding weight parameters. For example Figure 6 As shown, specifically, Figure 6 The figure shows a connection diagram of the selected input ends of two multi-select data selectors and the selected input ends and the output ends of the corresponding shift registers when the configurable data flow convolution module performs a point-by-point convolution operation. In this way, nine outputs can be obtained, and the data is then passed to the subsequent configurable addition tree module for accumulation to complete the point-by-point convolution operation.

[0064] For depthwise separable convolution, since two different types of convolution operations, depthwise convolution and pointwise convolution, are required, two dedicated data flow modules are often designed to adapt to the two types of convolution, but this inevitably increases the hardware resources and area overhead required for the design. The configurable data flow convolution module designed by the present invention can complete the convolution operations of pointwise convolution and depthwise convolution and different feature map input sizes by adopting different configurations, and fully reuses the feature map data, so that the module can be reused to the maximum extent, saving hardware resources and reducing the design area. In addition, the computing unit (multiplier) can be turned off or on by an external input switch signal, which also reduces the power consumption generated by unnecessary calculations.

[0065] In depthwise separable convolution, in addition to the different convolution data flow modes, the corresponding accumulation operations of point-by-point convolution and depthwise convolution are also different. For depthwise convolution, the input channel is 1, so there is no accumulation in the channel direction. The convolution kernel size is generally 3*3, so the accumulated input in two-dimensional space is 9, such as Figure 7 As shown in , for a depthwise convolution with N outputs in parallel, one convolution operation obtains N cumulative results in two-dimensional space. For point-by-point convolution, the convolution kernel size is 1*1, so there is no accumulation in two-dimensional space, but its input channel is often large, and it is necessary to perform accumulation operations in the direction of the input channel, such as Figure 8 As shown, for a point-by-point convolution with N input channels in parallel (the point-by-point convolution input channel parallelism is equal to the depth-wise convolution output channel parallelism), one convolution operation obtains the accumulated results on 9 (the number of output channels is 9) N input channels.

[0066] For different accumulation operations of depthwise convolution and pointwise convolution, the accumulation input of depthwise convolution is 9 inputs, and the accumulation input of pointwise convolution is equal to the parallel number N of input channels. In the design of accelerator, the parallel number N of pointwise convolution is usually 8, 16, or 32. Therefore, the present invention designs a configurable addition tree module, which can complete the accumulation operations of 8, 9, 16, and 32 inputs through configuration, and can simultaneously adapt to the accumulation requirements of different parallel numbers of input channels for depthwise convolution and pointwise convolution.

[0067] For example, Fig. 9 As shown, each configurable addition tree module includes: 4 8-input addition trees J1-J4, 5 2-input adders E1-E5 and 6 2-to-1 data selectors S1-S6. Fig. 9 As shown, the connection relationship between these devices is as follows:

[0068] The control ends of S1~S6 are all used to access external input selection control signals, and the input ends of J1~J4 are all used to access 8 groups of input data. The output end of J1 is connected to an input end of E1, the first input end of S2 and the second input end of S1. The other input end of E1 is used to access 1 group of data. The output end of E1 is connected to the first input end of S1, and the output end of S1 is the first output end O1 of the configurable addition tree module; the output end of S2 is connected to an input end of E2, the output end of J2 is connected to the other input end of E2 and the second input end of S3, the first input end of S3 is connected to an output end of E2, the output end of S3 is the second output end O2 of the configurable addition tree module, the other output end of E2 is connected to an input end of E3, and the other output end of E2 is the third output end O3 of the configurable addition tree module; the output end of J3 is connected to an input end of E4, the first input end of S4 The input ends are all connected, the other input end of E4 is connected to the output end of S5, one output end of E4 is connected to the second input end of S4, the other output end of E4 is connected to the other input end of E3, the output end of E3 is the fourth output end O4 of the configurable addition tree module, and the other output end of E4 is the fifth output end O5 of the configurable addition tree module, and the output end of S4 is the sixth output end O6 of the configurable addition tree module; the output end of J4 is connected to the second input end of S5, one input end of E5 and the first input end of S6, the second input end of S5 is used to access 1 group of data, the other input end of E5 is used to access 1 group of data, the output end of E5 is connected to the second input end of S6, and the output end of S6 is the seventh output end O7 of the configurable addition tree module.

[0069] Here, the input addition mode of the configurable addition tree module can be configured through the external input selection control signal connected to the control terminals of S1 to S6 and the inputs of J1 to J4, E1, S2, S5 and E5.

[0070] Specifically, for deep convolution, Fig. 9 and Fig.10 As shown by the blue line in , the left data selector selects 1 input data as output and adds it to the 8-input addition tree to complete the 9-input addition function, and then the right data selector selects the corresponding output result. A configurable addition tree module can output 4 9-input results at the same time; that is, Fig. 9 and Fig.10 As shown by the blue line in , when the first input of S1, the second input of S2, the first input of S3, the second input of S4, the first input of S5 and the second input of S6 are all selected, a configurable adder tree module works in 9-input addition mode, where O1, O2, O6 and O7 are all used to output a 9-input accumulation result. Therefore, for N-output parallel depth convolution, only A configurable addition tree module can complete all the accumulation operations to obtain the deep convolution accumulation result.

[0071] Specifically, for point-by-point convolution, when the input channel parallelism is 8, such as Fig. 9 and Fig.11 As shown by the yellow line in , the result of the 8-input addition tree output is directly transmitted to the data selector on the right for output. A configurable addition tree module can output 4 8-input results at the same time; that is, Fig. 9 and Fig.11 As shown by the yellow line in , when the second input of S1, the second input of S3, the first input of S4 and the first input of S6 are all enabled, and both inputs of S2 and S5 are not enabled, the adder tree module can be configured to work in 8-input addition mode, where O1, O2, O6 and O7 are all used to output an 8-input accumulation result. When the input channel parallelism is 16, as shown in Fig. 9 and Fig.12 As shown by the red line in , after the left data selector selects 8-input data as output, it is added to the output of the 8-input addition tree to complete the accumulation of 16 inputs, and then the result is directly output. A configurable addition tree module can output two 16-input results at the same time; that is, Fig. 9 and Fig.12 As shown by the red line in , when the first input of S2 and the second input of S5 are both enabled, and the two inputs of S1, S3, S4 and S6 are not enabled, the configurable adder tree module can work in 16-input addition mode, where O3 and O5 are used to output a 16-input accumulation result. When the input channel parallelism is 32, as shown in Fig. 9 and Fig.13 As shown by the purple line in , the 32-input addition function can be completed by simply adding the above two 16-input cumulative sums. A configurable addition tree module can output a 32-input result at the same time; that is, Fig. 9 and Fig.13 As shown by the purple line in , when the first input terminal of S2 and the second input terminal of S5 are both selected, and the two input terminals of S1, S3, S4 and S6 are not selected, the configurable addition tree module can also work in the 32-input addition mode, wherein O4 is used to output a 32-input accumulation result.

[0072] The configurable addition tree module designed in the present invention can be configured to complete the accumulation requirements of point-by-point convolution and depth-wise convolution, so that the two types of convolution can share the same addition tree unit, and there is also a flexible configuration for the point-by-point convolution input parallelism, which can flexibly adapt to different accumulation needs, greatly improving the utilization rate of the module, thereby greatly reducing the resources and area overhead required for the design.

[0073] It should be noted that the number of configurable data flow convolution modules and configurable addition tree modules in the above system, as well as the number of other modules can be set according to actual needs, and the present invention does not limit this.

[0074] The present invention also provides a configurable depth-separable convolution hardware acceleration method, which can be executed by the above-mentioned configurable depth-separable convolution hardware acceleration system, and the method includes.

[0075] S101. Obtain the first input feature map and weight parameters of the current depthwise separable convolutional network layer respectively; each depthwise separable convolutional network layer includes a depthwise convolutional layer and a cascaded point-by-point convolutional layer.

[0076] S102. Perform a deep convolution operation on the first input feature map in a data flow convolution mode of deep convolution to obtain a deep convolution result.

[0077] S103. Accumulate the depth convolution results in an input addition mode matching the depth convolution to obtain the depth convolution accumulation result.

[0078] S104: Process the deep convolution accumulation result to obtain an output feature map of the deep convolution layer, and cache the output feature map of the deep convolution layer.

[0079] S105. In the data flow convolution mode of point-by-point convolution, the output feature map of the depth convolution layer is used as the second input feature map, and in each round, a point-by-point convolution operation is performed on the data of c channels in the cached output feature map of the second input depth convolution layer to obtain the point-by-point convolution result of this round.

[0080] S106. Accumulate the results of each round of point-by-point convolution in an input addition mode that matches the point-by-point convolution.

[0081] S107, accumulating the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round, until all the accumulated point-by-point convolution results of all rounds are accumulated.

[0082] S108. Process the accumulated results of all rounds of point-by-point convolution results to obtain an output feature map of the point-by-point convolution layer, where the output feature map of the point-by-point convolution layer is the output feature map of the current depth-wise separable convolutional network layer, and cache the output feature map of the current depth-wise separable convolutional network layer to serve as the input feature map for calculation of the next network layer.

[0083] It should be noted that the specific processing principle of each step can refer to the above introduction to the structure and function of each module in the system, and the present invention will not be repeated here.

[0084] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0086] In the specification, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality of components. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good effects.

[0087] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A configurable depth-separable convolution hardware acceleration system, characterized in that: include: A feature map storage buffer and a weight storage buffer, used to store the first input feature map and weight parameters of the current depth-separable convolutional network layer respectively; A configurable data flow convolution module is used to perform a depth convolution operation on the first input feature map in a depth convolution data flow convolution mode to obtain a depth convolution result; in a point-by-point convolution data flow convolution mode, perform a point-by-point convolution operation on the data of c channels in the second input feature map of the point-by-point convolution layer in each round to obtain the point-by-point convolution result of this round; A configurable addition tree module is used to accumulate the depth convolution results in an input addition mode matching the depth convolution to obtain the depth convolution accumulation result; and accumulate each round of point-by-point convolution results in an input addition mode matching the point-by-point convolution; An intermediate result accumulation module is used to accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated; The accumulation result processing module is used to process the deep convolution accumulation result to obtain the output feature map of the deep convolution layer; process the accumulation result of all rounds of accumulated point-by-point convolution results to obtain the output feature map of the current deep separable convolution network layer; the output feature map of the deep convolution layer is the second input feature map.

2. The configurable depth-wise separable convolution hardware acceleration system according to claim 1, characterized in that: The system also includes: a point-by-point convolution intermediate result storage buffer, which is used to store the accumulated point-by-point convolution results of each round and the accumulated point-by-point convolution results of the previous round as accumulated intermediate results, and each time the intermediate result accumulation module performs an accumulation operation, the currently stored latest accumulated intermediate result is sent to the intermediate result accumulation module, and after the intermediate result accumulation module completes an accumulation operation, the updated accumulated intermediate result is received from the intermediate result accumulation module for storage.

3. The configurable depth-wise separable convolution hardware acceleration system according to claim 1, characterized in that: The configurable data flow convolution module is specifically used to work in a data flow convolution mode of depth convolution under the control of a first control signal, and perform a depth convolution operation on the first input feature map according to a weight parameter in the data flow convolution mode of depth convolution to obtain a depth convolution result; and, under the control of a second control signal, work in a data flow convolution mode of point-by-point convolution, and perform a point-by-point convolution operation on the data of c channels in the output feature map of the depth convolution layer according to the weight parameter in each round in the data flow convolution mode of point-by-point convolution to obtain the point-by-point convolution result of this round; wherein, when the output feature map of the depth convolution layer has X channels, the total number of rounds of point-by-point convolution operation is 4. The configurable depthwise separable convolution hardware acceleration system according to claim 1, characterized in that: Each configurable data flow convolution module includes: a shift register array, a register array, a multiplier array and two multiple-to-one data selectors; The shift register array has three rows, the first row and the second row have M shift registers, the third row has 3 shift registers, and the output ends and input ends of two adjacent shift registers in each row are connected in sequence; M is a preset positive integer greater than 3; each shift register is used to store part of the feature map data; The register array has three rows and three columns, each row and each column has three registers, and each register is used to store a part of the weight parameters; The multiplier array has three rows and three columns, and each row and each column has three multipliers; Among them, the output end of the jth shift register in the i-th row and the output end of the jth register in the i-th row are respectively connected to the two input ends of the jth multiplier in the i-th row, wherein i and j are both positive integers, and the values ​​of i and j are both 1 to 3, and the third input end of each multiplier is also used to access the external input switch signal, and a switch signal is used to control whether a corresponding multiplier is in a working state; the output end of the first multiple-choice data selector is connected to the input end of the first shift register in the second row, and the output end of the second multiple-choice data selector is connected to the input end of the first shift register in the third row; the output end of the first row of the shift register array is connected to the input end of the first shift register in the first row. The output end of each register is connected to the 1st to P+1th input ends of the first multiple-select-one data selector one by one, and the first The output ends of the registers are connected one by one with the 1st to P+1th input ends of the second multiple-choice data selector, wherein a is an integer, and the value of a is 0 to P, and P is a preset positive integer greater than or equal to 1, the output end of the third shift register in the first row of the shift register array is connected with the P+2th input end of the first multiple-choice data selector, and the output end of the third shift register in the second row of the shift register array is connected with the P+2th input end of the second multiple-choice data selector; the control ends of the first multiple-choice data selector and the second multiple-choice data selector are both used to access the same control signal, and the control signal is used to control the selected input end of the multiple-choice data selector so that the configurable data flow convolution module operates in the data flow convolution mode of depth convolution or the data flow convolution mode of point-by-point convolution.

5. The configurable depth-wise separable convolution hardware acceleration system according to claim 4, characterized in that: When the control signal is the first control signal, the 1st to Yth input terminals of the first multiple-choice-one data selector and the second multiple-choice-one data selector are sequentially selected, and the configurable data flow convolution module enters a data flow convolution mode of depthwise convolution, wherein Y is less than or equal to P+1; When the control signal is the second control signal, the P+2th input terminals of the first multi-select-one data selector and the second multi-select-one data selector are enabled, and the configurable data flow convolution module enters a data flow convolution mode of point-by-point convolution.

6. The configurable depth-wise separable convolution hardware acceleration system according to claim 1, characterized in that: The input addition modes of each configurable addition tree module include: 8-input addition mode, 9-input addition mode, 16-input addition mode and 32-input addition mode; In the 8-input addition mode, each configurable addition tree module is used to simultaneously output 4 8-input accumulation results; in the 9-input addition mode, each configurable addition tree module is used to simultaneously output 4 9-input accumulation results; in the 16-input addition mode, each configurable addition tree module is used to simultaneously output 2 16-input accumulation results; in the 32-input addition mode, each configurable addition tree module is used to simultaneously output 1 32-input accumulation result; wherein, the input addition mode matching the depthwise convolution is the 9-input addition mode, and the input addition mode matching the point-by-point convolution is any one of the 8-input addition mode, the 16-input addition mode and the 32-input addition mode.

7. The configurable depth-wise separable convolution hardware acceleration system according to claim 6, characterized in that: Each configurable addition tree module includes: 4 8-input addition trees J1-J4, 5 2-input adders E1-E5 and 6 2-to-1 data selectors S1-S6; The control terminals of S1 to S6 are used to access external input selection control signals, and the input terminals of J1 to J4 are used to access 8 sets of input data; The output end of J1 is connected to one input end of E1, the first input end of S2 and the second input end of S1, the other input end of E1 is used to access a group of data, the output end of E1 is connected to the first input end of S1, and the output end of S1 is the first output end O1 of the configurable addition tree module; The output end of S2 is connected to an input end of E2, the output end of J2 is connected to the other input end of E2 and the second input end of S3, the first input end of S3 is connected to an output end of E2, the output end of S3 is the second output end O2 of the configurable addition tree module, the other output end of E2 is connected to an input end of E3, and the other output end of E2 is the third output end O3 of the configurable addition tree module; The output end of J3 is connected to one input end of E4 and the first input end of S4, the other input end of E4 is connected to the output end of S5, one output end of E4 is connected to the second input end of S4, the other output end of E4 is connected to the other input end of E3, the output end of E3 is the fourth output end O4 of the configurable addition tree module, and the other output end of E4 is the fifth output end O5 of the configurable addition tree module, and the output end of S4 is the sixth output end O6 of the configurable addition tree module; The output end of J4 is connected to the second input end of S5, one input end of E5 and the first input end of S6. The second input end of S5 is used to access a group of data. The other input end of E5 is used to access a group of data. The output end of E5 is connected to the second input end of S6. The output end of S6 is the seventh output end O7 of the configurable addition tree module.

8. The configurable depthwise separable convolution hardware acceleration system according to claim 7, characterized in that: When the second input terminal of S1, the second input terminal of S3, the first input terminal of S4 and the first input terminal of S6 are all enabled, and both input terminals of S2 and S5 are not enabled, the configurable addition tree module operates in an 8-input addition mode, wherein O1, O2, O6 and O7 are all used to output an 8-input accumulation result; When the first input terminal of S1, the second input terminal of S2, the first input terminal of S3, the second input terminal of S4, the first input terminal of S5 and the second input terminal of S6 are all selected, the configurable addition tree module operates in a 9-input addition mode, wherein O1, O2, O6 and O7 are all used to output a 9-input accumulation result; When the first input terminal of S2 and the second input terminal of S5 are both enabled, and the two input terminals of S1, S3, S4 and S6 are not enabled, the configurable addition tree module operates in a 16-input addition mode and a 32-input addition mode, wherein O3 and O5 are both used to output a 16-input accumulation result, and O4 is used to output a 32-input accumulation result.

9. The configurable depth-wise separable convolution hardware acceleration system according to claim 1, characterized in that: When the current depth-separable convolutional network layer is the first layer of the neural network processed by the system, the feature map storage buffer is specifically used to obtain the first input feature map from the off-chip storage and send it to the configurable data flow convolution module; receive the output feature map of the depth convolution layer sent by the cumulative result processing block and store it; send the data of c channels in the stored output feature map of the depth convolution layer to the configurable data flow convolution module in each round to perform this round of point-by-point convolution operation; and store the output feature map of the current depth-separable convolutional network layer sent by the cumulative result processing module, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depth-separable convolutional network layer as the input feature map of the next layer; When the current depth-separable convolutional network layer is the Zth layer of the neural network processed by the system, the feature map storage buffer is specifically used to use the output feature map of the Z-1th layer stored in itself as the first input feature map, and send it to the configurable data flow convolution module; receive the output feature map of the depth convolution layer sent by the accumulation result processing module and store it; in each round, send the data of c channels in the stored output feature map of the depth convolution layer to the configurable data flow convolution module to perform this round of point-by-point convolution operation, and when the system performs calculations on the next layer of the neural network, send the output feature map of the current depth-separable convolutional network layer as the input feature map of the next layer, where Z is a positive integer greater than 1.

10. A configurable depth-separable convolution hardware acceleration method, characterized in that: Applied to the configurable depthwise separable convolution hardware acceleration system as described in any one of claims 1 to 9 above, the method comprises: The first input feature map and weight parameters of the current depth-separable convolutional network layer are obtained respectively; each depth-separable convolutional network layer includes a depth-wise convolutional layer and a cascaded point-by-point convolutional layer; Performing a deep convolution operation on the first input feature map in a data flow convolution mode of deep convolution to obtain a deep convolution result; Accumulating the depth convolution results in an input addition mode matching the depth convolution to obtain a depth convolution accumulation result; Processing the deep convolution accumulation result to obtain an output feature map of the deep convolution layer, and caching the output feature map of the deep convolution layer; In the data flow convolution mode of point-by-point convolution, the output feature map of the depth convolution layer is used as the second input feature map, and the data of c channels in the cached second input feature map are subjected to point-by-point convolution operation in each round to obtain the point-by-point convolution result of this round; Accumulate the results of each round of point-by-point convolution in an input addition mode that matches the point-by-point convolution; Accumulate the accumulated point-by-point convolution results of each round with the accumulated point-by-point convolution results of the previous round until all the accumulated point-by-point convolution results of all rounds are accumulated; The accumulated results of all rounds of accumulated point-by-point convolution results are processed to obtain an output feature map of the point-by-point convolution layer, where the output feature map of the point-by-point convolution layer is the output feature map of the current depth-separable convolutional network layer, and the output feature map of the current depth-separable convolutional network layer is cached to serve as an input feature map for calculation of the next network layer.

Images