Quantitative lightweight neural network hardware accelerator and design method thereof

Abstract

The invention belongs to the technical field of embedded hardware acceleration and computer vision, and relates to a quantitative lightweight neural network hardware accelerator and a design method thereof. The hardware accelerator comprises a top-layer control module, a memory control module, a reconfigurable memory, an input data preprocessing module, a core computing unit and a post-processing module; the memory control module controls the reconfigurable memory to send data information to the top-layer control module, the input data preprocessing module and the post-processing module; the input data preprocessing module is used for data and weight preparation; and the core calculation unit is used for completing data preparation, weight preparation and post-processing through a multiplier, an adder tree and an accumulator array. The hardware accelerator has higher computing performance, hardware resource utilization rate and data reuse rate; according to the method, the problem of low storage data utilization rate of the input and output cache is solved; hardware operator fusion of a convolution operator and a pooling operator is realized, and storage delay and access power consumption of data reading and writing are reduced.

Description

Technical Field

[0001] The invention belongs to the technical fields of hardware acceleration design, embedded design and computer vision, and relates to a quantized lightweight neural network hardware accelerator and a design method thereof. Background Art

[0002] Deep neural networks have become the core technology of many important applications, especially in the fields of computer vision and speech processing. Traditional models are often unable to cope with these complex tasks, while deep neural networks have shown good modeling capabilities. Convolutional neural networks are inspired by biological visual perception mechanisms and are particularly suitable for computer vision tasks such as image recognition and image segmentation. In recent years, many different CNN architectures have been proposed to improve the performance of models in terms of computational efficiency and recognition accuracy. However, these performance improvements are often accompanied by a significant increase in computing, storage, and bandwidth loads. For example, the ResNet network achieved a top-1 accuracy of 82% and a top-5 accuracy of 96.5% on the ImageNet dataset, but this achievement came at the cost of huge computing and memory overhead. This means that the network can only run in systems with strong computing power and sufficient storage space, and it is difficult to deploy on small mobile devices with limited resources. To solve this problem, a series of efficient and lightweight convolutional neural network architectures such as MobileNet have emerged. They have reduced the computational burden and model parameter size by redesigning the network structure, and have achieved certain success. However, despite the performance improvements of these efficient networks, many edge devices still face bottlenecks in storage and other aspects, and further breakthroughs and improvements are urgently needed.

[0003] With the development of industrial Internet of Things and the Internet, there is a demand for real-time operation of high-precision, low-complexity neural network models in edge embedded devices. However, due to the cost, area, power consumption and other practical issues of these embedded devices, their computing and storage capabilities are very limited. For example, they cannot be equipped with low-power, cheap, large-capacity off-chip storage (such as DRAM) to store the parameters required for convolutional neural network calculations, nor can they be equipped with a large number of multipliers to speed up the multiplication and accumulation operations in convolutional neural network calculations, thus failing to meet the real-time requirements of most applications. In addition, different computing units of neural networks have different computing modes, and designing independent computing units for different computing modes is a waste of resources. Therefore, how to make full use of smaller high-speed storage and weaker computing power on edge embedded devices to realize real-time neural network applications has become a difficult problem. Summary of the invention

[0004] The purpose of the present invention is to propose a quantized lightweight neural network hardware accelerator and a design method thereof to address the technical defects of the current neural network platform, which is limited in resources and cannot be deployed on low-power, small-area embedded edge devices.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] As a first aspect of the present invention, a quantized lightweight neural network hardware accelerator is proposed, comprising a top-level control module, a memory control module, a reconfigurable memory, an input data preprocessing module, a core computing unit, and a post-processing module. The top-level control module is responsible for controlling the entire hardware accelerator, and sends configuration information of corresponding units or modules to the input data preprocessing module, the core computing unit, and the post-processing module, and transmits configuration and feedback information bidirectionally with the memory control module; the memory control module controls the reconfigurable memory to send data information to the top-level control module, the input data preprocessing module, and the post-processing module; the input data preprocessing module comprises a data preparation submodule and a weight preparation submodule; the core The core computing unit includes a multiplier array, an adder tree array and an accumulator array and receives data from a data preprocessing module to a data preparation submodule, a weight preparation submodule and a post-processing module; the data preparation submodule includes a first submodule and a second submodule; the first submodule includes a row buffer and a register group for preparing input data for channel-by-channel convolution and standard ordinary convolution; the second submodule includes a register group for preparing input data for point-by-point convolution and fully connected operators; the core computing unit includes a multiplier array, an adder tree array and an accumulator array; the post-processing module includes a quantization submodule, a pooling submodule and a cross-layer connection submodule; the cross-layer connection submodule includes an address generation submodule.

[0007] The memory control module includes an input data memory control submodule, a weight memory control submodule and a quantization memory control submodule; the input data memory control submodule contains the read and write control logic of the first memory and the second memory; the weight memory control submodule and the quantization memory control submodule respectively contain the read and write control logic of the third memory and the fourth memory.

[0008] The reconfigurable memory includes a first memory, a second memory, a third memory and a fourth memory which are independent of each other; the first memory and the second memory are input data memory, the third memory and the fourth memory are weight parameter memory and quantization parameter memory respectively, which store different parameters required in neural network calculation respectively; the reading and writing of the first memory and the second memory are controlled by an input data memory control submodule, and the memory control module can complete the input data rearrangement and partial zero padding to adapt to the calculation requirements of the core computing unit; the reading and writing of the third memory and the fourth memory are controlled by a weight memory control submodule and a quantization memory control submodule respectively.

[0009] The input data preprocessing module includes a data preparation submodule and a weight preparation submodule; the data preparation submodule is connected to the first memory, the second memory and the multiplier array inside the core computing unit; the weight preparation submodule receives the data from the third memory and gives it to the multiplier array inside the core computing unit after processing; the multiplier array multiplies the received data and then performs addition and accumulation operations in sequence through the adder tree array and the accumulator array.

[0010] The pooling submodule of the post-processing module receives the output data of the data preparation submodule in the input data pre-processing module, performs pooling processing on the output data, and then inputs the pooled data into the quantization submodule.

[0011] The output data of the accumulator array, the fourth memory and the pooling submodule are all sent to the quantization submodule for quantization processing. The control information generated by the quantization processing is sent to the cross-layer connection submodule for cross-layer connection processing, and the generated address is returned to the memory control module.

[0012] The functions of the components in the hardware accelerator are as follows:

[0013] The top-level control module receives an external input control signal to start the hardware accelerator, determines the current state of the hardware accelerator, and notifies the input data memory control submodule to determine the read and write logic of the first memory and the second memory according to the current state; notifies the weight memory control submodule to read out the weight parameters according to the calculation requirements; notifies the quantization memory control submodule to read out the quantization parameters according to the quantization calculation requirements; notifies the data preparation submodule to receive the output data from the first memory or the second memory and perform zero padding and data rearrangement according to the calculation requirements; notifies the weight preparation submodule to receive the output data from the third memory and prepares the weight data according to the calculation requirements; notifies the core computing unit to receive the output from the data preparation submodule and the weight preparation submodule to complete the convolution calculation; notifies the post-processing module to enable the pooling submodule, the quantization submodule and the cross-layer connection submodule according to the calculation requirements.

[0014] The input data memory control submodule sends the output data of the first memory or the second memory to the data preparation submodule according to the configuration information of the top-level control module; the weight memory control submodule sends the output data of the third memory to the weight preparation submodule according to the configuration information of the top-level control module.

[0015] The data preparation submodule receives data output from the first memory or the second memory according to the configuration information of the top-level control module. The first submodule is responsible for converting the data into the data format required for channel-by-channel convolution and standard ordinary convolution, and the second submodule is responsible for converting the data into the data format required for point-by-point convolution and fully connected calculation; the weight preparation submodule receives data output from the third memory according to the configuration information of the top-level control module, fills the weight data with zeros and routes the data according to the current calculation requirements, and implements ping-pong cache switching for the weight data.

[0016] The hardware accelerator uses a storage utilization method different from the prior art, specifically:

[0017] (1) The memory control module is connected to the first, second, third and fourth memories, thereby realizing phased storage of data;

[0018] (2) The data preprocessing module, with the assistance of the output data and addresses of the cross-layer connection module, relies on an efficient zero-padding and data rearrangement mechanism for neural networks to achieve data rearrangement and zero-padding of the calculated feature graph;

[0019] The core computing unit receives the output data of the data preparation submodule and the weight preparation submodule according to the configuration information of the top-level control module, and changes the organization of the multiplier array, the adder tree array and the accumulator array according to the configuration information to complete one of the three convolution operators: standard ordinary convolution, channel-by-channel convolution, point-by-point convolution and full connection.

[0020] The hardware accelerator solves the problem of low utilization of input and output caches of the same size when storing data under traditional designs due to the imbalance in input and output sizes of standard ordinary convolution, channel-by-channel convolution, and point-by-point convolution. In specific implementation, data preparation is achieved by relying on data staged storage and data preprocessing modules.

[0021] The post-processing module enables the pooling submodule, quantization submodule and cross-layer connection submodule according to the configuration information of the top-level control module. The pooling submodule decides whether to enable it according to whether the current calculation is a pooling operator. The pooling submodule receives the output data from the core computing unit and completes the pooling calculation; the quantization submodule receives the output data from the core computing unit or the pooling submodule according to the configuration information of the top-level control module and completes the quantization calculation; the cross-layer connection submodule receives the output data of the quantization submodule according to the configuration information of the top-level control module and rearranges the data, converting the data arrangement format into the data format required for the next calculation; the address generation submodule is connected to the input data memory control submodule, and is responsible for writing the calculation results back to the first memory or the second memory.

[0022] Therefore, the computing structure used by the hardware accelerator realizes the hardware operator fusion of the convolution operator and the pooling operator of the convolution calculation, reducing the storage delay and access power consumption caused by repeated reading and writing of data; in specific implementation, it is realized through the interaction of data, address and control of the core computing unit and the pooling submodule;

[0023] As another aspect of the present invention, a design method for a quantized lightweight neural network hardware accelerator is proposed, comprising the following steps:

[0024] S1, receiving external input configuration information and calculating quantization parameters and weights based on the configuration information, and outputting the configuration instructions contained in the configuration information and the calculated weights and quantization parameters;

[0025] S2, receiving the configuration instruction and the calculated weight and quantization parameters, and sending the weight and quantization parameters to the corresponding storage location according to the configuration instruction;

[0026] S3, receiving external input instruction data and sending the instruction data to the instruction buffer;

[0027] S4, decoding the instruction data in the instruction buffer, and performing memory control, data preprocessing, core calculation and post-processing on the decoded instructions;

[0028] S5, the instruction buffer sends the next instruction data according to the configuration information, stores it in the instruction buffer, and then jumps to S4 to execute the instruction data;

[0029] S6. Repeat S5 until the calculation of the last instruction is completed, and then return the data input into the data memory to the external storage.

[0030] S4 The memory control is specifically: controlling the reading address and reading logic of the input data memory, weight parameter memory and quantization parameter memory according to the received instruction information; the data preprocessing is specifically to rearrange and fill the input data with zeros according to the received instruction information; the post-processing includes pooling, quantization and cross-layer connection; the pooling is specifically: determining whether to enable pooling according to the received instruction; if the current instruction indicates that pooling needs to be enabled, the pooling operator is calculated; otherwise, if the current instruction does not need to enable pooling, pooling is not enabled, and the input data is quantized as output data; the quantization is specifically: performing quantization calculations on the quantization parameters output by the quantization parameter memory according to the received instruction to obtain the quantized data; the cross-layer connection is specifically: performing cross-layer connection operations on the quantized data according to the received instruction to generate a write address for the input data memory.

[0031] The core calculation is specifically as follows: receiving parallel input data and weight parameter zero-filled data and arranging the received instructions into different forms to complete different types of calculations; the parallel input data is the data received by the parallel input data storage device, which is rearranged and zero-filled according to the external input instruction data; the weight parameter zero-filled data is the data output after the data in the weight parameter storage device is zero-filled.

[0032] The different types of calculations are implemented through a multiplier array, an adder tree array, and an accumulator array, which are specifically determined by the received instructions, and standard ordinary convolution, channel-by-channel convolution, point-by-point convolution, or full-connection calculations are performed according to the received instruction information. In specific implementation: the method performs parallel calculations in different dimensions based on different convolution operators, which has higher computing performance, hardware resource utilization, and data reuse rate compared to conventional loop expansion methods.

[0033] Beneficial Effects

[0034] The hardware accelerator for quantized lightweight neural network and the design method thereof proposed in the present invention have the following beneficial effects compared with the existing neural network acceleration methods:

[0035] 1. The storage utilization method used by the hardware accelerator solves the problem of low utilization of input and output caches of the same size in traditional designs when storing data due to imbalanced input and output sizes of standard ordinary convolution, channel-by-channel convolution, and point-by-point convolution;

[0036] 2. The hardware accelerator adopts an efficient zero-padding and data rearrangement mechanism for neural networks, and realizes data rearrangement and zero-padding of the computational feature graph with the help of an input data preprocessing module and a cross-layer connection module;

[0037] 3. The computing structure used by the hardware accelerator realizes the hardware operator fusion of the convolution operator and the pooling operator of the convolution calculation, reducing the storage delay and access power consumption caused by repeated reading and writing of data;

[0038] 4. The method proposes that different operators are expanded in different dimensions for parallel computing. Compared with the conventional loop expansion method, it has higher computing performance, hardware resource utilization and data reuse rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0040] Figure 1 A schematic diagram of the composition and connection relationship of a quantized lightweight neural network hardware accelerator and a design method thereof according to the present invention;

[0041] Figure 2 A flowchart of a quantized lightweight neural network hardware accelerator and a design method thereof according to the present invention;

[0042] Figure 3 The present invention provides a circuit connection diagram of a quantized lightweight neural network hardware accelerator and a design method thereof. DETAILED DESCRIPTION

[0043] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0044] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0045] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The following at least one item (items) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (items) of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0046] Example 1

[0047] As a first aspect of the present invention, a quantized lightweight neural network hardware accelerator is proposed, and the combination and connection block diagram is as follows: Figure 1 As shown, it includes a top-level control module, a memory control module, a reconfigurable memory, an input data preprocessing module, a core computing unit, and a post-processing module. The top-level control module is responsible for controlling the entire hardware accelerator, and sends the configuration information of the corresponding unit or module to the input data preprocessing module, the core computing unit, and the post-processing module, and transmits configuration and feedback information bidirectionally with the memory control module; the memory control module controls the reconfigurable memory to send data information to the top-level control module, the input data preprocessing module, and the post-processing module; the input data preprocessing module includes a data preparation submodule and a weight preparation submodule; the core computing unit includes a multiplier array, The adder tree array and the accumulator array receive data from the data preprocessing module to the data preparation submodule, the weight preparation submodule and the post-processing module; the data preparation submodule includes a first submodule and a second submodule; the first submodule includes a row buffer and a register group for preparing input data for channel-by-channel convolution and standard ordinary convolution; the second submodule includes a register group for preparing input data for point-by-point convolution and fully connected operators; the core computing unit includes a multiplier array, an adder tree array and an accumulator array; the post-processing module includes a quantization submodule, a pooling submodule and a cross-layer connection submodule; the cross-layer connection submodule includes an address generation submodule.

[0048] The memory control module includes an input data memory control submodule, a weight memory control submodule and a quantization memory control submodule; the input data memory control submodule contains the read and write control logic of the first memory and the second memory; the weight memory control submodule and the quantization memory control submodule respectively contain the read and write control logic of the third memory and the fourth memory.

[0049] The reconfigurable memory includes a first memory, a second memory, a third memory and a fourth memory which are independent of each other; the first memory and the second memory are input data memory, the third memory and the fourth memory are weight parameter memory and quantization parameter memory respectively, which store different parameters required in neural network calculation respectively; the reading and writing of the first memory and the second memory are controlled by an input data memory control submodule, and the memory control module can complete the input data rearrangement and partial zero padding to adapt to the calculation requirements of the core computing unit; the reading and writing of the third memory and the fourth memory are controlled by a weight memory control submodule and a quantization memory control submodule respectively.

[0050] The input data preprocessing module includes a data preparation submodule and a weight preparation submodule; the data preparation submodule is connected to the first memory, the second memory and the multiplier array inside the core computing unit; the weight preparation submodule receives the data from the third memory and gives it to the multiplier array inside the core computing unit after processing; the multiplier array multiplies the received data and then performs addition and accumulation operations in sequence through the adder tree array and the accumulator array.

[0051] The pooling submodule of the post-processing module receives the output data of the data preparation submodule in the input data pre-processing module, performs pooling processing on the output data, and then inputs the pooled data into the quantization submodule.

[0052] The output data of the accumulator array, the fourth memory and the pooling submodule are all sent to the quantization submodule for quantization processing. The control information generated by the quantization processing is sent to the cross-layer connection submodule for cross-layer connection processing, and the generated address is returned to the memory control module.

[0053] The functions of the components in the hardware accelerator are as follows:

[0054] The top-level control module receives an external input control signal to start the hardware accelerator, determines the current state of the hardware accelerator, and notifies the input data memory control submodule to determine the read and write logic of the first memory and the second memory according to the current state; notifies the weight memory control submodule to read out the weight parameters according to the calculation requirements; notifies the quantization memory control submodule to read out the quantization parameters according to the quantization calculation requirements; notifies the data preparation submodule to receive the output data from the first memory or the second memory and perform zero padding and data rearrangement according to the calculation requirements; notifies the weight preparation submodule to receive the output data from the third memory and prepares the weight data according to the calculation requirements; notifies the core computing unit to receive the output from the data preparation submodule and the weight preparation submodule to complete the convolution calculation; notifies the post-processing module to enable the pooling submodule, the quantization submodule and the cross-layer connection submodule according to the calculation requirements.

[0055] The input data memory control submodule sends the output data of the first memory or the second memory to the data preparation submodule according to the configuration information of the top-level control module; the weight memory control submodule sends the output data of the third memory to the weight preparation submodule according to the configuration information of the top-level control module.

[0056] The data preparation submodule receives data output from the first memory or the second memory according to the configuration information of the top-level control module. The first submodule is responsible for converting the data into the data format required for channel-by-channel convolution and standard ordinary convolution, and the second submodule is responsible for converting the data into the data format required for point-by-point convolution and fully connected calculation; the weight preparation submodule receives data output from the third memory according to the configuration information of the top-level control module, fills the weight data with zeros and routes the data according to the current calculation requirements, and implements ping-pong cache switching for the weight data.

[0057] The hardware accelerator uses a storage utilization method different from the prior art, specifically:

[0058] (1) The memory control module is connected to the first, second, third and fourth memories, thereby realizing phased storage of data;

[0059] (2) The data preprocessing module, with the assistance of the output data and addresses of the cross-layer connection module, relies on an efficient zero-padding and data rearrangement mechanism for neural networks to achieve data rearrangement and zero-padding of the calculated feature graph;

[0060] The core computing unit receives the output data of the data preparation submodule and the weight preparation submodule according to the configuration information of the top-level control module, and changes the organization of the multiplier array, the adder tree array and the accumulator array according to the configuration information to complete one of the three convolution operators: standard ordinary convolution, channel-by-channel convolution, point-by-point convolution and full connection.

[0061] The hardware accelerator solves the problem of low utilization of input and output caches of the same size when storing data under traditional designs due to the imbalance in input and output sizes of standard ordinary convolution, channel-by-channel convolution, and point-by-point convolution. In specific implementation, data preparation is achieved by relying on data staged storage and data preprocessing modules.

[0062] The post-processing module enables the pooling submodule, quantization submodule and cross-layer connection submodule according to the configuration information of the top-level control module. The pooling submodule decides whether to enable it according to whether the current calculation is a pooling operator. The pooling submodule receives the output data from the core computing unit and completes the pooling calculation; the quantization submodule receives the output data from the core computing unit or the pooling submodule according to the configuration information of the top-level control module and completes the quantization calculation; the cross-layer connection submodule receives the output data of the quantization submodule according to the configuration information of the top-level control module and rearranges the data, converting the data arrangement format into the data format required for the next calculation; the address generation submodule is connected to the input data memory control submodule, and is responsible for writing the calculation results back to the first memory or the second memory.

[0063] Therefore, the computing structure used by the hardware accelerator realizes the hardware operator fusion of the convolution operator and the pooling operator of the convolution calculation, reducing the storage delay and access power consumption caused by repeated reading and writing of data; in specific implementation, it is realized through the interaction of data, address and control of the core computing unit and the pooling submodule;

[0064] In specific implementation, the first submodule is used for input data preparation of channel-by-channel convolution and common convolution, including a row buffer and a register group; the second submodule is used for point-by-point convolution and full connection operator, mainly including a register group;

[0065] The multiplier array is composed of a combination of multipliers, and can form multipliers of different specifications according to the configuration. The multiplier is used as a common multiplier for all calculation operators, the upper level is an input data preprocessing module, and the lower level is a post-processing module;

[0066] The configurable adder tree array adds the outputs of the multiplication array according to different channel parallelisms according to the parallel number of input and output channels configured by the top-level control module, thereby achieving configurable input and output channel parallelism.

[0067] The accumulator module can be configured into different forms of accumulators according to the top-level control module, so as to achieve the accumulation function for different operators;

[0068] The post-processing module includes a quantization submodule, a pooling submodule and a cross-layer connection submodule;

[0069] The cross-layer connection submodule includes an address generation submodule;

[0070] The quantization submodule satisfies the output of different parallelisms according to the configuration of the top-level control module;

[0071] The pooling submodule contains FIFO and comparator, which is responsible for calculating the pooling operator;

[0072] The connection relationship between the components in the hardware accelerator is as follows:

[0073] The top-level control module is connected to the memory control module, the core computing unit and the post-processing module; the memory control module is connected to the top-level control module, the core computing unit and the post-processing module; the core computing unit is connected to the top-level control module, the memory control module and the post-processing module; the post-processing module is connected to the top-level control module, the memory control module and the core computing unit.

[0074] The connection mode inside the core computing unit is that the input data preprocessing module, the multiplier array, the adder tree array and the accumulator array are connected in sequence.

[0075] The input data preprocessing module is connected to the input data memory 1, the input data memory 2 and the multiplier array.

[0076] The internal connection mode of the multiplier array is that the input data preprocessing module, the weight parameter memory and the multiplier are connected.

[0077] The internal connection mode of the post-processing module is that the quantization module, the quantization parameter memory, the pooling submodule, the cross-layer connection submodule and the address generation submodule are connected.

[0078] As another aspect of the present invention, a design method for a quantized lightweight neural network hardware accelerator is proposed. Figure 2 As shown, the following steps are included:

[0079] S1, receiving external input configuration information and calculating quantization parameters and weights based on the configuration information, and outputting the configuration instructions contained in the configuration information and the calculated weights and quantization parameters;

[0080] S2, receiving the configuration instruction and the calculated weight and quantization parameters, and sending the weight and quantization parameters to the corresponding storage location according to the configuration instruction;

[0081] S3, receiving external input instruction data and sending the instruction data to the instruction buffer;

[0082] S4, decoding the instruction data in the instruction buffer, and performing memory control, data preprocessing, core calculation and post-processing on the decoded instructions;

[0083] S5, the instruction buffer sends the next instruction data according to the configuration information, stores it in the instruction buffer, and then jumps to S4 to execute the instruction data;

[0084] S6. Repeat S5 until the calculation of the last instruction is completed, and then return the data input into the data memory to the external storage.

[0085] S4 The memory control is specifically: controlling the reading address and reading logic of the input data memory, weight parameter memory and quantization parameter memory according to the received instruction information; the data preprocessing is specifically to rearrange and fill the input data with zeros according to the received instruction information; the post-processing includes pooling, quantization and cross-layer connection; the pooling is specifically: determining whether to enable pooling according to the received instruction; if the current instruction indicates that pooling needs to be enabled, the pooling operator is calculated; otherwise, if the current instruction does not need to enable pooling, pooling is not enabled, and the input data is quantized as output data; the quantization is specifically: performing quantization calculations on the quantization parameters output by the quantization parameter memory according to the received instruction to obtain the quantized data; the cross-layer connection is specifically: performing cross-layer connection operations on the quantized data according to the received instruction to generate a write address for the input data memory.

[0086] The core calculation is specifically as follows: receiving parallel input data and weight parameter zero-filled data and arranging the received instructions into different forms to complete different types of calculations; the parallel input data is the data received by the parallel input data storage device, which is rearranged and zero-filled according to the external input instruction data; the weight parameter zero-filled data is the data output after the data in the weight parameter storage device is zero-filled.

[0087] The different types of calculations are implemented through a multiplier array, an adder tree array, and an accumulator array, which are specifically determined by the received instructions, and standard ordinary convolution, channel-by-channel convolution, point-by-point convolution, or full-connection calculations are performed according to the received instruction information. In specific implementation: the method performs parallel calculations in different dimensions based on different convolution operators, which has higher computing performance, hardware resource utilization, and data reuse rate compared to conventional loop expansion methods.

[0088] S1, the top-level control module receives the external input configuration information and calculation parameters to calculate the weights and quantization parameters and outputs the configuration information instructions, weights and quantization parameters to the memory control module;

[0089] S2, the memory control module receives the configuration information, weights and quantization parameters from the top-level control module, and sends the weights and quantization parameters to corresponding storage locations according to the configuration information;

[0090] S3, the top-level control module receives the external input instruction data and sends it to the instruction buffer;

[0091] S4, the top-level control module decodes the instructions in the instruction buffer, and sends the decoded instructions to the memory control module, the input data preprocessing module, the core computing unit and the post-processing module;

[0092] The post-processing module includes a pooling submodule, a quantization submodule and a cross-layer connection submodule;

[0093] The pooling submodule in the post-processing module determines whether to enable the pooling module according to the received instruction;

[0094] Among them, if the current instruction indicates that the pooling submodule needs to be enabled, the submodule receives the output from the core computing unit and performs the calculation of the pooling operator; if the current instruction indicates that the pooling submodule is not needed, the submodule is not enabled, and the input data is directly sent as output data to the quantization submodule in the post-processing module;

[0095] The quantization submodule in the post-processing module performs quantization calculation according to the received instructions; receives the output from the core computing unit and the output from the quantization parameter memory to perform pooling calculation;

[0096] Step 10: the cross-layer connection submodule in the post-processing module performs a cross-layer connection operation according to the received instruction; the address generation submodule in the cross-layer connection submodule generates a write address for the input data storage according to the received instruction;

[0097] Step 11: The instruction buffer of the top-level control module sends the next instruction to each module according to the configuration information to calculate the next instruction;

[0098] Step 12: After the top-level control module completes the calculation of the last instruction according to the configuration information, the top-level control module returns the data input into the data memory to the external storage.

[0099] The method can achieve smaller on-chip storage occupancy and hardware resource consumption than traditional general-purpose neural network hardware accelerators in the hardware acceleration of a neural network that adopts a deep separable convolutional structure similar to the MobileNet network, analyze the characteristics of the convolutional neural network, and determine the convolution operation loop expansion method.

[0100] 1. Analysis of convolution kernel dimension (loop1). In embedded application scenarios, the size of the convolution kernel is 3×3 most of the time, the convolution kernel is small, and the moving step of the convolution kernel is also 1. In addition, it is relatively simple to expand the parallel computing control logic in the convolution kernel dimension, saving the hardware overhead of the control logic. Therefore, the hardware accelerator performs parallel computing in the convolution kernel dimension (loop1) when performing convolution operations.

[0101] 2. Analysis of input channel dimension (loop2) and output channel dimension (loop2). In embedded systems, the input channel and output channel of a convolutional neural network are usually between 1 and 256. Since the coupling between the input channel and the output channel is low, the control logic is simple, and the expansion of the input channel and the output channel is conducive to compatibility with more types of calculations. Therefore, the hardware accelerator performs parallel calculations on the input channel or the output channel when performing convolution operations.

[0102] 3. Analysis of output feature map dimension (loop3). Since expanding the output feature map loop will lead to complex data routing and control logic, the hardware accelerator does not perform parallel calculations in the output feature map dimension when performing convolution operations.

[0103] In summary, the hardware accelerator chooses to perform parallel computing in three dimensions: convolution kernel, input channel, and output channel, so as to improve the performance ceiling, simplify the data path, and improve the reuse rate of hardware resources.

[0104] Based on the loop expansion method proposed in this implementation, parallel calculations are carried out in the dimensions of convolution kernel and output channel for standard ordinary convolution and channel-by-channel convolution, corresponding to loop1-w, loop1-h and loop4 in the figure below; parallel calculations are carried out in the dimensions of input channel and output channel for point-by-point convolution and full connection, corresponding to loop2 and loop4 in the figure below.

[0105] for(no=0;no<;no++){loop4

[0106] for(y=0;y <Noy;y++){loop3-h

[0107] for(x=0;x <Nox;x++){loop3-w

[0108] for(ni=0;ni <Nif;ni++){loop2

[0109] for(ky=0;ky <Nky;ky++){loop1-h

[0110] for(kx=0;kx <Nkx;kx++){loop1-w

[0111] pixel L (no; x, y) + = pixel L-1 ×weight L-1

[0112] }}}}}}

[0113] For the case where pooling calculation follows standard ordinary convolution or channel-by-channel convolution, the hardware accelerator performs operator fusion at the hardware level. Before the hardware operator fusion, the calculation order is convolution calculation, quantization calculation, pooling calculation, and quantization calculation. After the hardware operator fusion, the calculation order is convolution calculation, pooling calculation, and quantization calculation, which greatly reduces the repeated reading and writing delay of data and the reading and writing power consumption.

[0114] In the specific implementation, the deployment platform of the hardware design is the Xilinx ARTIX-7FPGA development board. The specific deployment and connection method of the hardware module can be referred to Figure 1 . The neural network specifically used in this embodiment 1 is a lightweight convolutional neural network based on MobileNetV2, and its specific network structure and weight parameters are provided by the torchvision function inside Pytorch. The parameters of the network are quantized to 8 bits through post-training quantization to further reduce the computational complexity and resource consumption. In post-training quantization, the quantization range of the operation input and output is 0 to 255, the quantization range of the network weight is -128 to 127, and the quantization ranges of the quantization parameters M, N, and Z are 0 to 65535, 0 to 255, and -128 to 127, respectively. Among them, the quantization method of the network weight adopts symmetric quantization. The use of this quantization method can make the hardware calculate the quantization result without considering the additional multiplication and accumulation operation caused by the weight quantization zero point, so as to further reduce the computational burden and resource consumption.

[0115] The core computing unit of the hardware accelerator sets the number of multipliers to 18 parallel. For ordinary convolution, channel-by-channel convolution, and fully connected calculations, the number of multipliers used is 18, which can achieve 100% resource utilization. For point-by-point convolution, the number of multipliers used is 16, which can achieve 89% resource utilization. This number is the result of comprehensive consideration of the accelerator's power consumption requirements, storage bandwidth, design real-time requirements, and overall design frequency. The 18 parallel multipliers are expanded differently for different operators. For ordinary convolution and channel-by-channel convolution, 18 parallel is 9 parallel for the convolution kernel dimension and 2 parallel for the output channel; for point-by-point convolution, 16 parallel is 8 parallel for the input channel and 2 parallel for the output channel. At the same time, the outputs of different operators are all 2 parallel, which allows the subsequent pipelines to match, unifies the output form of the data, and reduces the occupation of hardware resources.

[0116] The on-chip storage system is divided into four parts, namely two input data memories, a weight parameter memory, and a quantization parameter memory. The size of these four memories is determined by the storage resources provided by the hardware platform and the size of the neural network model. In this embodiment 1, the size of the first memory and the second memory is 8KB, the size of the weight parameter memory is 12KB, and the size of the quantization parameter memory is 4KB. Among them, the first memory and the second memory can both be used as input data cache or output data cache. At the same time, different input data preprocessing modules are designed for different forms of operators. Compared with traditional input and output, using the same data path can increase data access speed and reduce system latency.

[0117] The cross-layer connection submodule in the post-processing module proposes a new data rearrangement method, which rearranges the output data of the result of the previous convolution calculation according to the instructions received by the cross-layer connection submodule. The address generation submodule of the cross-layer connection submodule generates the write-back address of the data, thereby realizing continuous processing of the data. The additional data access delay is reduced through the cross-layer connection submodule.

[0118] In this embodiment 1, the order of the above convolution operations is adjusted by comprehensively considering the computing resources, input and output storage, partial accumulation and storage, the condition that all data only access the external storage once, and the splitting and implementation possibility of the convolution operation. Since the earlier Loop-1 and Loop-2 are completed, the smaller the required partial accumulation and storage can be, and when the input or weight can be completely stored on the chip, only one external storage access is required, so loop1-loop2-loop3-loop4 is adjusted to loop1-loop3-loop2-loop4 respectively. By changing the calculation order, the size of the on-chip buffer can be reduced.

[0119] In this embodiment 1, the hardware accelerator as a whole is mounted on the peripheral bus of the Hummingbird E203 as an IP, and the generation of FPGA burning bitstream is implemented in Vivado. In the specific implementation of the on-chip system circuit connection, the IP core is encapsulated into an IP with the Hummingbird E203 peripheral interface ICB protocol. The PS end is the open source RISC-V MCU Hummingbird E203 IP core. The operation IP core is the hardware accelerator IP proposed in this patent. The connection relationship between each IP core is as follows: the MCU and the IP core are connected through the ICB bus interface. The specific circuit connection method can be referred to. Figure 3 .

[0120] Example 2

[0121] Based on the above-mentioned hardware accelerator design and the construction of the test platform, this embodiment 2 provides a hardware acceleration method for a lightweight neural network based on MobileNetV2.

[0122] In this implementation, the first step is to fill the quantized neural network weights and quantization parameters with zeros, rearrange and save them in the order of convolution operations. This step only needs to be performed once, and all subsequent operations are based on the weights and parameters in the same order.

[0123] The second step is to fill zeros and rearrange the data of the input original image;

[0124] The third step is to manually compile the lightweight neural network and convert each layer of convolution operations into binary instructions;

[0125] Step 4: The network weights, quantization parameters and instruction streams are input into the neural network hardware accelerator through the MCU and the ICB bus;

[0126] Step 5: Wait for the neural network hardware accelerator to complete the operation and return the result. For the specific process, please refer to Figure 2 .

[0127] So far, the hardware acceleration of the lightweight neural network based on MobileNetV2 in this embodiment has been completed through the first to fifth steps. After testing, this embodiment 2 can realize real-time data processing (up to 13.4FPS) at a frequency of 250KHz with 3326 lookup table resources on the FPGA development board, and the on-chip storage resources occupied are 32KB.

[0128] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0129] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the present invention and its equivalents.

Claims

1. A quantized lightweight neural network hardware accelerator, characterized in that: include: Top-level control module, memory control module, reconfigurable memory, input data preprocessing module, core computing unit, post-processing module. The top-level control module is responsible for controlling the entire hardware accelerator, sending configuration information of corresponding units or modules to the input data preprocessing module, core computing unit and post-processing module, and bidirectionally transmitting configuration and feedback information with the memory control module. The memory control module controls the reconfigurable memory to send data information to the top-level control module, the input data preprocessing module and the post-processing module; the input data preprocessing module includes a data preparation submodule and a weight preparation submodule; The core computing unit includes a multiplier array, an adder tree array and an accumulator array and receives data from the data preprocessing module to the data preparation submodule, the weight preparation submodule and the post-processing module; the data preparation submodule includes a first submodule and a second submodule; The first submodule includes a row buffer and a register group for preparing input data for channel-by-channel convolution and standard ordinary convolution; the second submodule includes a register group for preparing input data for point-by-point convolution and fully connected operators; the core computing unit includes a multiplier array, an adder tree array, and an accumulator array; The post-processing module includes a quantization submodule, a pooling submodule and a cross-layer connection submodule; The cross-layer connection submodule includes an address generation submodule.

2. A quantized lightweight neural network hardware accelerator according to claim 1, characterized in that: The memory control module includes an input data memory control submodule, a weight memory control submodule and a quantization memory control submodule; the input data memory control submodule contains the read and write control logic of the first memory and the second memory; the weight memory control submodule and the quantization memory control submodule respectively contain the read and write control logic of the third memory and the fourth memory.

3. A quantized lightweight neural network hardware accelerator according to claim 2, characterized in that: The reconfigurable memory includes a first memory, a second memory, a third memory and a fourth memory which are independent of each other; the first memory and the second memory are input data memory, the third memory and the fourth memory are weight parameter memory and quantization parameter memory respectively, which store different parameters required in neural network calculation respectively; the reading and writing of the first memory and the second memory are controlled by an input data memory control submodule, and the memory control module can complete the input data rearrangement and partial zero padding to adapt to the calculation requirements of the core computing unit; the reading and writing of the third memory and the fourth memory are controlled by a weight memory control submodule and a quantization memory control submodule respectively.

4. A quantized lightweight neural network hardware accelerator according to claim 1, characterized in that: The input data preprocessing module includes a data preparation submodule and a weight preparation submodule; the data preparation submodule is connected to the first memory, the second memory and the multiplier array inside the core computing unit; The weight preparation submodule receives the data from the third memory and processes it before sending it to the multiplier array inside the core computing unit; The multiplier array performs multiplication processing on the received data and then sequentially passes through the adder tree array and the accumulator array for addition and accumulation operations.

5. The quantized lightweight neural network hardware accelerator according to claim 1, characterized in that: The pooling submodule of the post-processing module receives the output data of the data preparation submodule in the input data pre-processing module, performs pooling processing on the output data, and then inputs the pooled data into the quantization submodule.

6. A quantized lightweight neural network hardware accelerator according to claim 1, characterized in that: The output data of the accumulator array, the fourth memory and the pooling submodule are all sent to the quantization submodule for quantization processing. The control information generated by the quantization processing is sent to the cross-layer connection submodule for cross-layer connection processing, and the generated address is returned to the memory control module.

7. A method for designing a quantized lightweight neural network hardware accelerator, used to design the quantized lightweight neural network hardware accelerator according to any one of claims 1 to 6, characterized in that: The steps include: S1, receiving external input configuration information and calculating quantization parameters and weights based on the configuration information, and outputting the configuration instructions contained in the configuration information and the calculated weights and quantization parameters; S2, receiving the configuration instruction and the calculated weight and quantization parameters, and sending the weight and quantization parameters to the corresponding storage location according to the configuration instruction; S3, receiving external input instruction data and sending the instruction data to the instruction buffer; S4, decoding the instruction data in the instruction buffer, and performing memory control, data preprocessing, core calculation and post-processing on the decoded instructions; S5, the instruction buffer sends the next instruction data according to the configuration information, stores it in the instruction buffer, and then jumps to S4 to execute the instruction data; S6. Repeat S5 until the calculation of the last instruction is completed, and then return the data input into the data memory to the external storage.

8. The method for designing a quantized lightweight neural network hardware accelerator according to claim 7, characterized in that: The memory control in S4 is specifically as follows: controlling the reading address and reading logic of the input data memory, weight parameter memory and quantization parameter memory according to the received instruction information; the data preprocessing is specifically to rearrange and fill zeros on the input data according to the received instruction information; the post-processing includes pooling, quantization and cross-layer connection; the cross-layer connection is to perform cross-layer connection operations on the quantized data according to the received instructions to generate a write address for the input data memory.

9. The method for designing a quantized lightweight neural network hardware accelerator according to claim 7, characterized in that: The core calculation is specifically: receiving parallel input data and weight parameter zero-filling data and arranging the received instructions into different forms to complete different types of calculations; the parallel input data is the data received by the parallel input data memory, which is rearranged and zero-filled according to the external input instruction data; The weight parameter zero-filled data is data output after the data in the weight parameter memory is filled with zeros.

10. The design method of a quantized lightweight neural network hardware accelerator according to claim 9, characterized in that: The different types of calculations are implemented through a multiplier array, an adder tree array, and an accumulator array, and are specifically determined by the received instructions. Standard ordinary convolution, channel-by-channel convolution, point-by-point convolution, or fully connected calculations are performed based on the received instruction information.

Images