An accelerator and acceleration method for a lightweight convolutional neural network

Abstract

The application discloses an accelerator and an acceleration method of a light-weight convolutional neural network, the application adopts a calculation window intercepting module to perform zero padding on the last row of feature map data corresponding to each convolutional layer in the neural network, and adopts a preset mode to rearrange and splice the processed feature map data, so that invalid windows in calculation are eliminated, the processing time of the calculation window intercepting module is reduced, data multiplexing is met, feature map window data acquisition is well matched, power consumption is greatly reduced, resource utilization is improved, and calculation speed is accelerated; when the special convolution calculation module is used for convolution operation processing on different types of convolutional layers in the neural network, a DSP splicing and separating operation is adopted for the corresponding convolution calculation units, so that two corresponding calculation results can be obtained through one-time processing, and calculation efficiency is greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of chip design technology, specifically relating to a lightweight convolutional neural network accelerator and acceleration method. Background Technology

[0002] With the continuous development of neural network technology, its applications in real life are becoming increasingly widespread, such as image classification, speech recognition, and autonomous driving. However, as the number of neural network layers increases, the number of network parameters also increases, leading to many challenges in its deployment on mobile devices. The enormous computational load results in excessively high latency during neural network inference, which is unacceptable in some application areas.

[0003] Currently, hardware acceleration of neural networks has received widespread attention and has seen significant development. However, traditional acceleration schemes are too simplistic, failing to develop separate acceleration plans for different types of convolution operations. Furthermore, resource utilization during convolution calculations is insufficient, resulting in substantial resource waste and low computational efficiency. In addition, conventional acceleration schemes often transfer the computation results of each layer to off-chip storage, which typically leads to high power consumption. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a lightweight convolutional neural network accelerator and acceleration method. The technical problem to be solved by this invention is achieved through the following technical solution:

[0005] In a first aspect, the present invention provides a lightweight convolutional neural network accelerator, comprising:

[0006] The control module is used to control the operation of each module in the accelerator according to the information it receives.

[0007] The calculation parameter storage module is used to receive and store the weight data and bias parameters of the current convolutional layer in the neural network, as well as the feature map data of the first convolutional layer in the neural network, which are output from off-chip storage, under the control of the control module; wherein, when the calculation parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer.

[0008] The calculation window truncation module, under the control of the control module, performs zero-padding on the last row of the feature map data of the current convolutional layer to obtain processed feature map data; rearranges and splices the processed feature map data using a preset method to obtain feature map window data; and rearranges the weight data of the current convolutional layer and truncates it to obtain weight window data.

[0009] A dedicated convolutional computation module, under the control of the control module, selects the corresponding convolutional computation unit according to the type of the current convolutional layer to perform corresponding convolutional operations on the feature map window data and weight window data of the current convolutional layer, and obtains the computational processing result corresponding to the current convolutional layer; wherein, DSP splicing and separation operations are used when performing the corresponding convolutional operations.

[0010] The computational data processing module is used, under the control of the control module, to process the computational processing result corresponding to the current convolutional layer and the corresponding bias parameter accordingly, so as to obtain the data processing result corresponding to the current convolutional layer.

[0011] The feature map data storage module is used to store the data processing results corresponding to the current convolutional layer under the control of the control module; and to transmit the data processing results corresponding to the current convolutional layer as the feature map data of the next convolutional layer to the calculation window truncation module.

[0012] The non-convolutional computation module, under the control of the control module, performs pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network.

[0013] The calculation result output module is used to transmit the final result to the off-chip storage.

[0014] In one embodiment of the present invention, the information received by the control module includes:

[0015] The data bus outputs module control commands and parameters of the current convolutional layer in the neural network.

[0016] In one embodiment of the present invention, the processed feature map data is rearranged and spliced ​​using a preset method to obtain feature map window data, including:

[0017] The processed feature map data corresponding to the current convolutional layer is rearranged using a reverse cross-row transmission method to obtain rearranged feature map data.

[0018] The data of two adjacent rows in the rearranged feature map data are spliced ​​and truncated to obtain the feature map window data.

[0019] In one embodiment of the present invention, the convolution calculation unit includes:

[0020] Standard convolution calculation unit, pointwise convolution calculation unit, and depthwise convolution calculation unit.

[0021] In one embodiment of the present invention, the computational processing result corresponding to the current convolutional layer and the corresponding bias parameter are processed accordingly to obtain the data processing result corresponding to the current convolutional layer, including:

[0022] The computational data processing module adds the computational processing result corresponding to the current convolutional layer to the corresponding bias parameter, performs function activation, quantization processing and residual connection processing on the added result, and outputs the data processing result corresponding to the current convolutional layer.

[0023] In one embodiment of the present invention, when the computational data processing module performs function activation, it uses the ReLU activation function for processing.

[0024] In one embodiment of the present invention, the feature map data storage module stores the data processing results corresponding to the current convolutional layer, including:

[0025] The feature map data storage module selects the region corresponding to the type of the current convolutional layer within the feature map data storage module and stores the data processing results corresponding to the current convolutional layer.

[0026] In one embodiment of the present invention, during the process of storing the data processing results corresponding to the current convolutional layer, the feature map data storage module stores the results of two adjacent rows of the data processing results consecutively.

[0027] Secondly, the present invention provides a method for accelerating lightweight convolutional neural networks, applied to an accelerator for lightweight convolutional neural networks. The accelerator includes: a control module, a computational parameter storage module, a computational window truncation module, a dedicated convolutional computation module, a computational data processing module, a feature map data storage module, a non-convolutional computation module, and a computational result output module. The method includes:

[0028] The control module receives module control commands output from the data bus, as well as the parameters of the current convolutional layer in the neural network.

[0029] The control module controls the computational parameter storage module to receive and store the weight data and bias parameters of the current convolutional layer in the neural network, as well as the feature map data of the first convolutional layer in the neural network, which are output from off-chip storage; wherein, when the computational parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer;

[0030] The control module controls the calculation window truncation module to pad the last row of the feature map data of the current convolutional layer with zeros to obtain the processed feature map data; the processed feature map data is then rearranged and spliced ​​using a preset method to obtain the feature map window data; and the weight data of the current convolutional layer is rearranged and truncated to obtain the weight window data.

[0031] The control module controls the dedicated convolution calculation module, and selects the corresponding convolution calculation unit to perform corresponding convolution operations on the feature map window data and weight window data of the current convolution layer according to the type of the current convolution layer, so as to obtain the calculation result corresponding to the current convolution layer; wherein, DSP splicing and separation operation is used when performing the corresponding convolution operation.

[0032] The control module controls the computational data processing module to process the computational processing result and the corresponding bias parameter of the current convolutional layer to obtain the data processing result of the current convolutional layer.

[0033] The control module controls the feature map data storage module to store the data processing results corresponding to the current convolutional layer; the data processing results corresponding to the current convolutional layer are used as the feature map data of the next convolutional layer and transmitted to the calculation window truncation module.

[0034] The control module controls the non-convolutional computation module to perform pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network.

[0035] The final result of the neural network is transmitted to the off-chip storage using the calculation result output module.

[0036] The beneficial effects of this invention are:

[0037] In the solution provided by this invention, a computational window truncation module is used to pad the last row of the feature map data corresponding to each convolutional layer in the neural network with zeros. The processed feature map data is then rearranged and spliced ​​using a preset method, eliminating invalid windows in the computation and reducing the processing time of the computational window truncation module. This satisfies data reuse while effectively matching the acquisition of feature map window data, greatly reducing power consumption and improving resource utilization while accelerating computation speed. Furthermore, a dedicated convolutional computation module uses DSP splicing and separation operations when selecting the corresponding convolutional computation unit for different types of convolutional layers in the neural network, enabling two corresponding computation results to be obtained in one processing step, which greatly improves computational efficiency. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of the structure of a lightweight convolutional neural network accelerator provided in an embodiment of the present invention;

[0039] Figure 2 A schematic diagram of the structure of each module of a lightweight convolutional neural network accelerator provided in an embodiment of the present invention;

[0040] Figure 3This diagram illustrates the traditional zero-padding method and the invalid windows generated during calculation window truncation.

[0041] Figure 4 This is a schematic diagram illustrating data reuse using the traditional zero-padding method.

[0042] Figure 5 This is a schematic diagram of the pre-padding process for a lightweight convolutional neural network accelerator provided in an embodiment of the present invention;

[0043] Figure 6 The image shows the effect of pre-padding processing for a lightweight convolutional neural network accelerator provided in an embodiment of the present invention.

[0044] Figure 7 This is a schematic diagram of traditional data flow.

[0045] Figure 8 This is a schematic diagram of the reverse cross-row data flow of an accelerator for a lightweight convolutional neural network provided in an embodiment of the present invention;

[0046] Figure 9 This is a schematic diagram showing the feature map window cutoff of a lightweight convolutional neural network accelerator provided in an embodiment of the present invention.

[0047] Figure 10 This is a schematic diagram of a traditional DSP calculation method;

[0048] Figure 11 This is a schematic diagram of the DSP splicing and separation operation of a lightweight convolutional neural network accelerator provided in an embodiment of the present invention;

[0049] Figure 12 This is a schematic diagram illustrating the steps of a lightweight convolutional neural network acceleration method provided in an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] This invention provides an accelerator for lightweight convolutional neural networks, such as... Figure 1 As shown, it may include:

[0053] The control module is used to control the operation of each module in the accelerator according to the information it receives.

[0054] The computational parameter storage module is used to receive and store the weight data and bias parameters of the current convolutional layer in the neural network, as well as the feature map data of the first convolutional layer in the neural network, which are output from off-chip storage, under the control of the control module; wherein, when the computational parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer.

[0055] The computation window truncation module, under the control of the control module, pads the last row of the feature map data of the current convolutional layer with zeros to obtain the processed feature map data; rearranges and concatenates the processed feature map data using a preset method to obtain the feature map window data; and rearranges the weight data of the current convolutional layer and truncates it to obtain the weight window data.

[0056] A dedicated convolutional computation module, under the control of the control module, selects the corresponding convolutional computation unit according to the type of the current convolutional layer to perform corresponding convolutional operations on the feature map window data and weight window data of the current convolutional layer, and obtains the computational processing result of the current convolutional layer; wherein, DSP splicing and separation operations are used when performing the corresponding convolutional operations.

[0057] The computational data processing module, under the control of the control module, processes the computational results and corresponding bias parameters of the current convolutional layer to obtain the data processing results of the current convolutional layer.

[0058] The feature map data storage module is used to store the data processing results corresponding to the current convolutional layer under the control of the control module; and to transmit the data processing results corresponding to the current convolutional layer as the feature map data of the next convolutional layer to the calculation window truncation module.

[0059] The non-convolutional computation module, under the control of the control module, performs pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network.

[0060] The calculation result output module is used to transmit the final result to off-chip storage.

[0061] In general, the working process of the lightweight convolutional neural network accelerator includes: the control module issues corresponding instructions based on the module control commands output from the data bus, controlling the computation window truncation module to process the data obtained by the computation parameter storage module to obtain feature map window data and weight window data; according to the type of the current convolutional layer, the control module selects the corresponding convolution computation unit to perform corresponding convolution operations on the feature map window data and weight window data to obtain the computation processing result of the current convolutional layer; then, the control module processes the computation processing result and corresponding bias parameters of the current convolutional layer to obtain the data processing result of the current convolutional layer and stores the data processing result in the feature map data storage module; the feature map data storage module uses the data processing result of the current convolutional layer as the feature map data of the next convolutional layer and transmits it to the computation window truncation module to realize the processing of each convolutional layer in the neural network; finally, the control module outputs the final result of the neural network based on the data processing result of the last convolutional layer; and the computation result output module transmits the final result to off-chip storage.

[0062] Figure 2 This is a schematic diagram of the various modules of a lightweight convolutional neural network accelerator provided in an embodiment of the present invention; for ease of understanding, the following is combined with... Figure 1 and Figure 2 This paper provides a detailed description of each module of a lightweight convolutional neural network accelerator proposed in this embodiment of the invention.

[0063] Control module

[0064] It is used to control the operation of each module in the accelerator according to the information it receives.

[0065] Specifically, the information received by the control module may include:

[0066] The data bus outputs module control commands and parameters of the current convolutional layer in the neural network. The parameters of the current convolutional layer may include the number and type of the current convolutional layer.

[0067] The main function of the control module is to receive the parameters of the current convolutional layer in the accelerator from the data bus, as well as the command to be executed. Then, the control module controls each module in the accelerator to perform corresponding work in sequence according to the received information.

[0068] Calculation parameter storage module

[0069] The main function of the computation parameter storage module is to receive and store the weight data and bias parameters output from off-chip storage for the calculation of convolutional layers; it can also store the feature map data of the first convolutional layer in the neural network.

[0070] Optionally, the computational parameter storage module can consist of a ping-pong RAM, with the two RAMs taking turns reading and writing data to reduce data loading time.

[0071] Calculate window capture module

[0072] The computation window truncation module, under the control of the control module, pads the last row of the feature map data of the current convolutional layer with zeros to obtain the processed feature map data; it then rearranges and splices the processed feature map data using a preset method to obtain the feature map window data; and finally, it rearranges the weight data of the current convolutional layer and truncates it to obtain the weight window data.

[0073] from Figure 2 As can be seen from the diagram, the computation window truncation module can include a feature map window truncation unit and a weight window truncation unit, which are responsible for trunculating feature map window data and weight window data, respectively.

[0074] For a diagram illustrating traditional zero-padding methods and invalid windows generated during calculation window truncation, please refer to [link / reference]. Figure 3 This method involves padding the feature map with zeros in the left and right columns and the top and bottom rows. However, this method inevitably generates invalid windows during data flow, increasing data preprocessing time and reducing accelerator efficiency.

[0075] Please refer to the diagram illustrating data reuse under the traditional zero-padding method. Figure 4 As can be seen, when performing convolution calculations with a stride of 1, the data in rows 2 and 3 of the feature map should be reused, and the same applies to the data in other rows. This can greatly reduce the power consumption generated by data transmission. However, because it will produce invalid windows, this solution is not the optimal solution.

[0076] To address the issue of invalid windows generated by traditional zero-padding methods, this invention provides a novel pre-padding process. Please refer to [link to relevant documentation]. Figure 5 The computation window cropping module pads the last row of each channel in the feature map data corresponding to the current convolutional layer with zeros to obtain the processed feature map data.

[0077] The pre-padding method provided in this embodiment of the invention only performs zero-padding on the last row of each channel in the feature map data corresponding to the current convolutional layer. During subsequent operations, zero-padding of the top row is achieved by setting the feature map data flow buffer to 0, and zero-padding of the left and right columns is achieved by pre-padding, thereby achieving the same zero-padding effect while avoiding the generation of invalid windows. Therefore, this pre-padding method can eliminate the generation of invalid windows during data flow.

[0078] Please refer to the effect diagram of the pre-padding process of the accelerator for a lightweight convolutional neural network provided in this embodiment of the invention. Figure 6 As shown in the diagram, when the feature map window data moves to the end of a row, data from the next row enters the window. At this point, the window is invalid. However, by enabling zero padding, the data entering the window in this cycle can be changed to 0, while the original data is stored in an intermediate register. This data is then moved into the window again for output calculation when the window moves to the next row. Setting the first row of the feature map data flow buffer to 0 saves n cycles of data input time compared to the traditional method. Pre-padding the left and right columns with zeros saves 2*n cycle times per channel compared to the traditional zero-padding method, where n is the feature map size. This significantly reduces the accelerator's data preprocessing time.

[0079] When performing calculation window segmentation, it is also very important to make full use of the input data. Figure 7 This is a schematic diagram of traditional data flow. In traditional methods, the beginning and end of adjacent rows are connected, enabling data reuse. However, the convolution operation used in this embodiment is a DSP splicing and separation operation, which can obtain two results in a single window's convolution calculation. This makes the traditional data flow method unsuitable for this embodiment. Therefore, it is crucial to find a way to splice two windows together while ensuring full reuse of input data and applicability to other modules of the accelerator. Based on this, this embodiment proposes a reverse cross-row data flow. For a schematic diagram of the reverse cross-row data flow, please refer to [link to schematic diagram]. Figure 8 The way data flows, such as Figure 8 As shown; the processed feature map data is rearranged and spliced ​​using a preset method to obtain feature map window data, which may include:

[0080] The processed feature map data corresponding to the current convolutional layer is rearranged using a reverse cross-row transfer method to obtain rearranged feature map data. When rearranging using the reverse cross-row transfer method, the flow buffer memory of the feature map data is set to 0, thereby achieving the effect of padding the top row of the feature map data with zeros.

[0081] The data in adjacent rows of the rearranged feature map data are concatenated and truncated to obtain the feature map window data. During the concatenation and truncation process, zero padding is used to pad the left and right columns of the feature map data.

[0082] Specifically, by Figure 8As can be seen, the data flow method designed in this invention, taking a step size of 1 as an example, involves simultaneously inputting data from two adjacent rows of the feature map, using a reverse cross-row transmission method, and then concatenating the 8-bit data from the fifth and fourth rows, the fourth and third rows, and the third and second rows into 16-bit data to obtain the feature map window data. This data is then convolved with the weight window data, and the convolution result is separated to obtain two results simultaneously. The processed feature map data is then processed according to the following... Figure 8 The data flow pattern shown is processed sequentially. Combined with the previously described operations of setting the feature map data flow buffer to zero and pre-filling, corresponding processing is performed to achieve data reuse while avoiding the generation of invalid windows. This splicing method yields a 16-bit window that can achieve... Figure 9 The two 8-bit windows shown achieve the same effect and allow for full reuse of input data, greatly improving the efficiency of the accelerator.

[0083] Dedicated convolution calculation module

[0084] The dedicated convolutional computation module, under the control of the control module, selects the corresponding convolutional computation unit according to the type of the current convolutional layer to perform corresponding convolutional operations on the feature map window data and weight window data of the current convolutional layer, and obtains the computational processing result of the current convolutional layer; among them, DSP splicing and separation operations are used when performing the corresponding convolutional operations.

[0085] Dedicated convolution calculation modules, such as Figure 2 As shown, it may include:

[0086] Standard convolution calculation unit, pointwise convolution calculation unit, and depthwise convolution calculation unit.

[0087] The control module, based on the type of the current convolutional layer, controls the corresponding convolutional computation unit in the dedicated convolutional computation module to perform corresponding convolutional operations on the feature map window data and weight window data of the current convolutional layer. In addition, the dedicated convolutional computation module can also use general-purpose addition tree units to accumulate the convolutional results; this is existing technology and will not be elaborated upon here.

[0088] Specifically, since pointwise convolution is a 1x1 convolution operation, in addition to parallel computation on the input and output channels, it also involves feature... Figure 2 Parallel computing strategies are also employed on the 3D plane to improve the computational efficiency of the accelerator.

[0089] Since depthwise convolution is a single-channel convolution, it is not necessary to accumulate the convolution results across the input channel dimension. Therefore, the main parallelism design tends to focus on increasing the parallelism of the output channel.

[0090] Since standard convolution uses parallel computation in both input and output channels, the convolution result can only be output to the computation data processing module after the input channel dimensions are accumulated. When applying it, the parallel parameters are configured according to the parameters of each layer of the network.

[0091] In addition to configuring different convolutional units for each type of convolution, the dedicated convolutional computation module employs DSP splicing and separation operations during the convolution operation of each convolutional unit. Traditional convolution operations, such as... Figure 10 As shown, the computational precision of a DSP is typically 25*18 bits, while the requantized feature map data and weight data are both 8 bits, resulting in significant resource waste and extremely low DSP inference efficiency. This invention, based on the characteristics of convolution operations and the properties of computational resources, concatenates the feature map input data, such as... Figure 11 As shown, two multiplications with equal weights are calculated in a single DSP operation. Two 8-bit input data are concatenated into a 25-bit register with zero padding in between. Then, they are multiplied by the 8-bit weights to obtain a 32-bit output. Finally, a separation operation is performed to obtain the product of the two input data and their respective weights. Figure 11 If the output is 1, it represents the product of the lower 8 bits of the feature map input data and the weights. For the product of the higher 8 bits and the weights, only the 16th bit of the output needs to be checked. If it's 1, then adding 1 to the output 2 gives the result of the higher 8 bits; if it's 0, then output 2 is the product of the higher 8 bits. This data concatenation and separation method can achieve twice the efficiency while using the same hardware resources, and saves half the convolution time, greatly enhancing the accelerator's efficiency.

[0092] Computational data processing module

[0093] The computational data processing module, under the control of the control module, processes the computational results and corresponding bias parameters of the current convolutional layer to obtain the data processing results of the current convolutional layer.

[0094] Specifically, the computational results and corresponding bias parameters of the current convolutional layer are processed accordingly to obtain the data processing results for the current convolutional layer, which may include:

[0095] The computational data processing module adds the computational result of the current convolutional layer to the corresponding bias parameter, performs function activation, quantization, and residual connection processing on the added result, and outputs the data processing result of the current convolutional layer.

[0096] Specifically, the ReLU activation function is used when the computational data processing module performs function activation.

[0097] The main function of the computational data processing module is to activate the 32-bit data after convolution and accumulation using the ReLU function and then requantize it to 8 bits, such as... Figure 2 As shown, it consists of an activation function unit, a requantization unit, and a residual connection unit. Specifically, the activation function unit implements the ReLU activation function on the data output from the dedicated convolutional unit. The requantization unit requantizes the 32-bit input data to 8 bits. The residual connection unit is used to implement the residual connection operations required by the neural network. It adds the feature map data from a previous layer to the current feature map data via instructions from the control module before outputting it to the feature map data storage module or the non-convolutional module to complete the final computation of the neural network.

[0098] Feature map data storage module

[0099] The feature map data storage module, under the control of the control module, stores the data processing results corresponding to the current convolutional layer; and transmits the data processing results corresponding to the current convolutional layer as the feature map data of the next convolutional layer to the calculation window truncation module.

[0100] Specifically, the feature map data storage module stores the data processing results corresponding to the current convolutional layer, which may include:

[0101] The feature map data storage module selects the region corresponding to the type of the current convolutional layer from its internal settings and stores the data processing results of the current convolutional layer.

[0102] In the process of storing the data processing results corresponding to the current convolutional layer, the feature map data storage module stores the results of two adjacent rows of the data processing results consecutively.

[0103] The main function of the feature map data storage module is to store the feature map data of the intermediate layers of the network. After the feature map data is processed by convolution, it is first sent to the computation data processing module for processing according to the instructions of the control module, and then sent to the feature map data storage module. This avoids reading and writing the feature map data of each layer from outside the chip, reduces the power consumption generated by data transmission, and speeds up the computation efficiency.

[0104] Non-convolutional computation module

[0105] Under the control of the control module, the non-convolutional computation module performs pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network.

[0106] Non-convolutional computation modules, such as Figure 2As shown, the non-convolutional computation module mainly performs operations on the pooling layers and fully connected layers of the network. In neural networks, these two layers are usually located in the last few layers. Therefore, the output data of the non-convolutional computation module is fed into the computation result output module and stored on the external chip as the final result of the network, completing the hardware acceleration of the entire neural network.

[0107] Calculation result output module

[0108] The calculation result output module transmits the final result to off-chip storage.

[0109] The lightweight convolutional neural network accelerator proposed in this invention, after receiving network parameters, uses a control module to send instructions to control other modules to work sequentially. The calculation parameter storage module stores weight data and bias parameters transmitted from external sources. Ping-pong RAM reads and writes data in turn, reducing data loading time. Then, the calculation data is passed to the calculation window extraction module. The feature map extraction window unit uses pre-padding zeros. Zero padding in the top row and the left and right columns is achieved by setting the feature map data flow buffer to 0 and pre-padding zeros, respectively. This zero-padding method eliminates the generation of invalid windows during data flow, reducing data preprocessing time. The data flow method designed in this invention, by simultaneously inputting data from two adjacent rows of the feature map and using a reverse cross-row transmission method, adapts to DSP splicing and separation operations while allowing for full reuse of input data, greatly improving the accelerator's efficiency.

[0110] The dedicated convolution calculation module proposed in this embodiment of the invention not only designs a dedicated convolution unit for each type of convolution, but also adopts DSP splicing and separation operation. This data splicing and separation method can achieve twice the efficiency while using the same hardware resources, and saves half of the convolution time, greatly enhancing the efficiency of the accelerator.

[0111] Example 2

[0112] This invention provides a method for accelerating lightweight convolutional neural networks, such as... Figure 12 As shown, an accelerator for a lightweight convolutional neural network may include: a control module, a computation parameter storage module, a computation window truncation module, a dedicated convolution computation module, a computation data processing module, a feature map data storage module, a non-convolution computation module, and a computation result output module; the acceleration method for this lightweight convolutional neural network may include the following steps:

[0113] S1, using the control module to receive control commands from the data bus output module, as well as the parameters of the current convolutional layer in the neural network;

[0114] S2 uses the control module to control the computational parameter storage module, receiving and storing the weight data and bias parameters of the current convolutional layer in the neural network output from off-chip storage, as well as the feature map data of the first convolutional layer in the neural network; wherein, when the computational parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer;

[0115] S3, using the control module to control the calculation window truncation module, pads the last row of the feature map data of the current convolutional layer with zeros to obtain the processed feature map data; rearranges and splices the processed feature map data using a preset method to obtain the feature map window data; rearranges the weight data of the current convolutional layer and truncates it to obtain the weight window data.

[0116] S4 uses the control module to control the dedicated convolution calculation module. According to the type of the current convolutional layer, the corresponding convolution calculation unit is selected to perform the corresponding convolution operation on the feature map window data and weight window data of the current convolutional layer to obtain the calculation result of the current convolutional layer. Among them, DSP splicing and separation operation is used when performing the corresponding convolution operation.

[0117] Specifically, in S4, convolution multiplication is accelerated using DSP splicing and separation operations, achieving twice the efficiency while using the same hardware resources, and saving half of the convolution time, greatly enhancing the efficiency of the accelerator.

[0118] S5, using the control module to control the computational data processing module, processes the computational processing result and the corresponding bias parameter of the current convolutional layer accordingly to obtain the data processing result of the current convolutional layer;

[0119] S6, using the control module to control the feature map data storage module, stores the data processing result corresponding to the current convolutional layer; and uses the data processing result corresponding to the current convolutional layer as the feature map data of the next convolutional layer, and transmits it to the calculation window truncation module.

[0120] S7 uses the control module to control the non-convolutional computation module, performs pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network.

[0121] S8 uses the calculation result output module to transmit the final result of the neural network to off-chip storage.

[0122] The method provided in this embodiment can be applied to the accelerator provided in Embodiment 1 above, as detailed in Example 1. This accelerator can not only reduce the power consumption of convolutional neural networks and reduce resource waste, but also greatly improve the computing speed.

[0123] In the solution provided by this invention, a computational window truncation module is used to pad the last row of the feature map data corresponding to each convolutional layer in the neural network with zeros. The processed feature map data is then rearranged and spliced ​​using a preset method, eliminating invalid windows in the computation and reducing the processing time of the computational window truncation module. This satisfies data reuse while effectively matching the acquisition of feature map window data, greatly reducing power consumption and improving resource utilization while accelerating computation speed. Furthermore, a dedicated convolutional computation module uses DSP splicing and separation operations when selecting the corresponding convolutional computation unit for different types of convolutional layers in the neural network, enabling two corresponding computation results to be obtained in one processing step, which greatly improves computational efficiency.

[0124] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0125] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. An accelerator for a lightweight convolutional neural network, characterized in that, include: The control module is used to control the operation of each module in the accelerator according to the information it receives. The calculation parameter storage module is used to receive and store the weight data and bias parameters of the current convolutional layer in the neural network, as well as the feature map data of the first convolutional layer in the neural network, which are output from off-chip storage, under the control of the control module; wherein, when the calculation parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer. The calculation window truncation module, under the control of the control module, performs zero-padding on the last row of the feature map data of the current convolutional layer to obtain processed feature map data; rearranges and splices the processed feature map data using a preset method to obtain feature map window data; and rearranges the weight data of the current convolutional layer and truncates it to obtain weight window data. A dedicated convolutional computation module, under the control of the control module, selects the corresponding convolutional computation unit according to the type of the current convolutional layer to perform corresponding convolutional operations on the feature map window data and weight window data of the current convolutional layer, and obtains the computational processing result corresponding to the current convolutional layer; wherein, DSP splicing and separation operations are used when performing the corresponding convolutional operations. The computational data processing module is used, under the control of the control module, to process the computational processing result corresponding to the current convolutional layer and the corresponding bias parameter accordingly, so as to obtain the data processing result corresponding to the current convolutional layer. The feature map data storage module is used to store the data processing results corresponding to the current convolutional layer under the control of the control module; and to transmit the data processing results corresponding to the current convolutional layer as the feature map data of the next convolutional layer to the calculation window truncation module. The non-convolutional computation module, under the control of the control module, performs pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network. The calculation result output module is used to transmit the final result to the off-chip storage.

2. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, The information received by the control module includes: The data bus outputs module control commands and parameters of the current convolutional layer in the neural network.

3. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, The processed feature map data is rearranged and spliced ​​using a preset method to obtain feature map window data, including: The processed feature map data corresponding to the current convolutional layer is rearranged using a reverse cross-row transmission method to obtain rearranged feature map data. The data of two adjacent rows in the rearranged feature map data are spliced ​​and truncated to obtain the feature map window data.

4. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, The convolution calculation unit includes: Standard convolution calculation unit, pointwise convolution calculation unit, and depthwise convolution calculation unit.

5. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, The step of processing the computation result and the corresponding bias parameter of the current convolutional layer to obtain the data processing result of the current convolutional layer includes: The computational data processing module adds the computational processing result corresponding to the current convolutional layer to the corresponding bias parameter, performs function activation, quantization processing and residual connection processing on the added result, and outputs the data processing result corresponding to the current convolutional layer.

6. The lightweight convolutional neural network accelerator according to claim 5, characterized in that, When the computational data processing module performs function activation, it uses the ReLU activation function.

7. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, The feature map data storage module stores the data processing results corresponding to the current convolutional layer, including: The feature map data storage module selects the region corresponding to the type of the current convolutional layer within the feature map data storage module and stores the data processing results corresponding to the current convolutional layer.

8. The lightweight convolutional neural network accelerator according to claim 1, characterized in that, During the process of storing the data processing results corresponding to the current convolutional layer, the feature map data storage module stores the results of two adjacent rows of the data processing results consecutively.

9. A method for accelerating lightweight convolutional neural networks, characterized in that, An accelerator for lightweight convolutional neural networks, comprising: a control module, a computation parameter storage module, a computation window truncation module, a dedicated convolution computation module, a computation data processing module, a feature map data storage module, a non-convolution computation module, and a computation result output module; the method comprises: The control module receives module control commands output from the data bus, as well as the parameters of the current convolutional layer in the neural network. The control module controls the computational parameter storage module to receive and store the weight data and bias parameters of the current convolutional layer in the neural network, as well as the feature map data of the first convolutional layer in the neural network, which are output from off-chip storage; wherein, when the computational parameter storage module processes for the first time, the current convolutional layer is the first convolutional layer; The control module controls the calculation window truncation module to pad the last row of the feature map data of the current convolutional layer with zeros to obtain the processed feature map data; the processed feature map data is then rearranged and spliced ​​using a preset method to obtain the feature map window data; and the weight data of the current convolutional layer is rearranged and truncated to obtain the weight window data. The control module controls the dedicated convolution calculation module, and selects the corresponding convolution calculation unit to perform corresponding convolution operations on the feature map window data and weight window data of the current convolution layer according to the type of the current convolution layer, so as to obtain the calculation result corresponding to the current convolution layer; wherein, DSP splicing and separation operation is used when performing the corresponding convolution operation. The control module controls the computational data processing module to process the computational processing result and the corresponding bias parameter of the current convolutional layer to obtain the data processing result of the current convolutional layer. The control module controls the feature map data storage module to store the data processing results corresponding to the current convolutional layer; the data processing results corresponding to the current convolutional layer are used as the feature map data of the next convolutional layer and transmitted to the calculation window truncation module. The control module controls the non-convolutional computation module to perform pooling and fully connected processing on the data processing results corresponding to the last convolutional layer, and outputs the final result of the neural network. The final result of the neural network is transmitted to the off-chip storage using the calculation result output module.

Images