Neural network accelerator system for image super-resolution and implementation method thereof

Abstract

The application discloses a kind of neural network accelerator systems for image super-resolution and implementation method thereof, including transmission engine, logic control module, data cache module, computing complex, off-chip data bus, off-chip control bus, on-chip data bus and on-chip control bus, transmission engine and computing complex are connected with data cache module by on-chip data bus, transmission engine, data cache module and computing complex are connected with logic control module by on-chip control bus, transmission engine is also connected with off-chip data bus, logic control module is also connected with off-chip control bus, and computing complex includes data routing unit, data compression processor, matrix unit, convolution unit and activation unit.The embodiment of the application realizes the parallel acceleration calculation of convolution calculation, improves the calculation efficiency of hardware accelerator, reduces hardware resource consumption, and can be widely applied in neural network technical field.

Description

Technical Field

[0001] This invention relates to the field of neural network technology, and in particular to a neural network accelerator system for image super-resolution and its implementation method. Background Technology

[0002] Super-resolution (SMR) aims to recover a high-resolution version of an image from a corresponding low-resolution version. This task is a typical and challenging area in computer vision and image signal processing, and it has wide applications in the real world, such as image segmentation, remote sensing, and surveillance. With the increasing prevalence of high-definition and ultra-high-definition display devices, SMR has attracted growing attention. Meanwhile, in most scenarios, the speed of image SMR algorithms is crucial.

[0003] With the tremendous success of deep learning across various fields, deep learning-based super-resolution algorithms, such as convolutional neural networks, generative adversarial networks, and recurrent neural networks, have been extensively studied. In particular, when performing super-resolution on videos, these algorithms use a large number of low-resolution and high-resolution video sequences as input to the neural network, performing frame alignment, feature extraction, feature fusion, and super-resolution reconstruction. Due to the powerful non-linear learning capabilities of neural networks, deep learning-based super-resolution methods typically achieve good performance on many common standard image datasets. However, due to their high computational cost, large feature map cache, and high bandwidth requirements, they are difficult to perform in real-time on many embedded devices.

[0004] Most existing hardware accelerators fail to fully and effectively utilize the parallelism in convolution operations, resulting in reduced computational efficiency and increased hardware resource consumption. Furthermore, the use of variable-length coding network compression methods, such as Huffman coding, to increase the effective bandwidth of accelerators can lead to data misalignment at the hardware level, causing the computational pattern to lose its original regularity. Summary of the Invention

[0005] To address the aforementioned technical problems, the present invention aims to provide a high-efficiency neural network accelerator system for image super-resolution and its implementation method.

[0006] The first technical solution adopted in this invention is:

[0007] A neural network accelerator system for image super-resolution includes a transmission engine, a logic control module, a data cache module, a computational complex, an off-chip data bus, an off-chip control bus, an on-chip data bus, and an on-chip control bus. The transmission engine and the computational complex are both connected to the data cache module via the on-chip data bus. The transmission engine, the data cache module, and the computational complex are all connected to the logic control module via the on-chip control bus. The transmission engine is also connected to the off-chip data bus and is used for data transmission between the on-chip and off-chip caches. The logic control module is also connected to the off-chip control bus and is used for global control based on received control commands. The data cache module is used to cache feature map data and weight data. The computational complex is used to perform convolution computation tasks and includes a data routing unit, a data compression processor, a matrix unit, a convolution unit, and an activation unit. The data compression processor is used to compress / decompress the feature map data.

[0008] Furthermore, the data routing unit is used to rearrange the feature map data stored in the data caching module and then transmit it to the data compression processor for decompression. The matrix unit is used to generate multiple parallel convolution outputs based on the decompressed feature map data. The convolution unit is used to multiply and accumulate the parallel convolution outputs and the weight data to obtain the convolution calculation result. The activation unit is used to fit the convolution calculation result through a preset activation function to obtain the convolution-calculated feature map data, and then compress it through the data compression processor before transmitting it to the data caching module for storage.

[0009] Furthermore, the data compression processor includes a compression unit and a decompression unit. The compression unit includes a comparator tree, a first cache unit, a parallel subtraction unit, and a parallel division unit. The comparator tree is used to find the maximum and minimum values ​​in a compressed block and transmit them to the first cache unit. The parallel subtraction unit and the parallel division unit are used to perform compression processing based on the data cached by the first cache unit. The decompression unit includes a second cache unit, a subtraction unit, a parallel multiplication unit, and a parallel addition unit. The second cache unit is used to store the maximum and minimum values ​​in a compressed block. The subtraction unit, the parallel multiplication unit, and the parallel addition unit are used to perform decompression processing based on the data cached by the second cache unit.

[0010] Furthermore, the data compression processor uses the DXT-5 algorithm for compression / decompression processing.

[0011] Furthermore, the matrix unit includes a shift register group, a cross-group buffer circuit, and a timing control circuit. The cross-group buffer circuit includes multiple FIFO buffers. The timing control circuit is used to control the shift register group and the cross-group buffer circuit, so that the shift register group performs a shift operation or is set from its corresponding cross-group buffer circuit.

[0012] Furthermore, the convolutional unit includes multiple multiply-accumulate units, each including a control circuit, a dynamic shifter, a register, and a multiplier. The dynamic shifter is used to shift and align the input weight data and feature map data, and the register and the multiplier are used to perform multiply-accumulate calculations on the shifted and aligned weight data and feature map data.

[0013] Furthermore, the transmission engine includes a read DMA engine and a write DMA engine.

[0014] Furthermore, the logic control module includes an instruction cache unit, an instruction decoding unit, an input FSM, a convolutional FSM, and an output FSM.

[0015] Furthermore, the data caching module includes an input caching unit, an output caching unit, and a weight caching unit.

[0016] The second technical solution adopted in this invention is:

[0017] A method for implementing a neural network accelerator system for image super-resolution, comprising the following steps:

[0018] The feature map data and weight data are transmitted to the data cache module via the transmission engine.

[0019] The logic control unit acquires and decodes control commands, and then performs global control based on the decoding results.

[0020] The feature map data stored in the data cache module is rearranged by the data routing unit and then transmitted to the data compression processor.

[0021] The feature map data after data rearrangement is decompressed by the data compression processor and then transmitted to the matrix unit.

[0022] The matrix unit generates multiple parallel convolutional outputs based on the decompressed feature map data and transmits them to the convolutional unit.

[0023] The convolution unit performs multiplication and accumulation based on the parallel convolution output and the weight data stored in the data cache module to obtain the convolution calculation result, which is then transmitted to the activation unit.

[0024] The activation unit performs fitting processing on the convolution calculation result using a preset activation function to obtain the convolutional feature map data and transmits it to the data compression processor.

[0025] The feature map data after convolution calculation is compressed by the data compression processor and then transmitted to the data cache module.

[0026] The beneficial effects of this invention are: This invention provides a neural network accelerator system and its implementation method for image super-resolution. By using a customizable computational complex, the degree of data reuse within the accelerator is improved, parallel acceleration of convolution computation is achieved, the computational efficiency of the hardware accelerator is improved, and the consumption of hardware resources is reduced. Attached Figure Description

[0027] Figure 1 A schematic diagram of a neural network accelerator system for image super-resolution provided in an embodiment of the present invention;

[0028] Figure 2 This is a schematic diagram of the structure of the computational complex provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of a matrix unit provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of the structure of the multiply-accumulate unit provided in an embodiment of the present invention;

[0031] Figure 5 This is a flowchart illustrating the steps of an implementation method for a neural network accelerator system for image super-resolution, provided in an embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0033] In the description of this invention, "multiple" means two or more. The use of "first" and "second" is for distinguishing technical features only and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or the order in which the indicated technical features are presented. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this specification is for the purpose of describing particular embodiments only and not for limiting the invention.

[0034] Reference Figure 1 This invention provides a neural network accelerator system for image super-resolution, including a transmission engine, a logic control module, a data cache module, a computational complex, an off-chip data bus, an off-chip control bus, an on-chip data bus, and an on-chip control bus. The transmission engine and the computational complex are both connected to the data cache module via the on-chip data bus. The transmission engine, data cache module, and computational complex are all connected to the logic control module via the on-chip control bus. The transmission engine is also connected to the off-chip data bus and is used for data transmission between the on-chip and off-chip caches. The logic control module is also connected to the off-chip control bus and is used for global control based on received control commands. The data cache module is used to cache feature map data and weight data. The computational complex is used to perform convolution computation tasks and includes a data routing unit, a data compression processor, a matrix unit, a convolution unit, and an activation unit. The data compression processor is used to compress / decompress the feature map data.

[0035] Specifically, the transmission engine handles data transfer between the accelerator's on-chip and off-chip caches; the configurable logic control module decodes input control commands and performs global control; the data cache module stores feature map data, weight data, etc.; and the computational complex undertakes the majority of the accelerator's computational tasks. The computational complex includes a data routing unit, a data compression processor, a matrix unit, an activation unit, and a convolutional unit. The data compression processor implements an improved DXT-5 compression algorithm, performing feature map data compression and decompression on the hardware side to improve the accelerator's effective bandwidth. The remaining parts are responsible for the remaining steps in the convolutional computation.

[0036] As a further optional implementation, the data routing unit is used to rearrange the feature map data stored in the data caching module and then transmit it to the data compression processor for decompression processing. The matrix unit is used to generate multiple parallel convolution outputs based on the decompressed feature map data. The convolution unit is used to multiply and accumulate the parallel convolution outputs and weight data to obtain the convolution calculation result. The activation unit is used to fit the convolution calculation result through a preset activation function to obtain the convolution calculation feature map data, and then compress it through the data compression processor before transmitting it to the data caching module for storage.

[0037] Specifically, firstly, the feature map data in the data caching module is sent to the data routing unit for data rearrangement to accommodate zero padding under different convolution kernel sizes. After data rearrangement, it is sent to the data compression processor for data decompression. Then, the matrix unit, mainly composed of shift register groups, is responsible for generating the output of a certain number of parallel convolution windows and sending them to the convolution unit for multiplication and accumulation with the weights. The activation unit that follows is responsible for applying activation functions to the feature map. Finally, the data compression unit in the data compression processor is responsible for compressing the feature map data and sending it back to the data caching module.

[0038] As a further optional implementation, the data compression processor includes a compression unit and a decompression unit. The compression unit includes a comparator tree, a first cache unit, a parallel subtraction unit, and a parallel division unit. The comparator tree is used to find the maximum and minimum values ​​in a compressed block and transmit them to the first cache unit. The parallel subtraction unit and the parallel division unit are used to perform compression processing based on the data cached in the first cache unit. The decompression unit includes a second cache unit, a subtraction unit, a parallel multiplication unit, and a parallel addition unit. The second cache unit is used to store the maximum and minimum values ​​in a compressed block. The subtraction unit, the parallel multiplication unit, and the parallel addition unit are used to perform decompression processing based on the data cached in the second cache unit.

[0039] Specifically, the data compression processor includes a decompression unit and a compression unit. In the decompression unit, to implement the decompression operation in the DXT-5 image compression algorithm, a second buffer unit, a subtraction unit, a parallel multiplication unit, and a parallel addition unit are designed. The second buffer unit stores the maximum and minimum values ​​in a compressed block. The subtraction, parallel multiplication, and parallel addition units implement the decompression algorithm to recover the data from the compressed index values. In the compression unit, to implement the compression operation, a comparator tree, a first buffer unit, a parallel subtraction unit, and a parallel division unit are designed. The comparator tree finds the maximum and minimum values ​​in a compressed block and stores them in the first buffer unit. The parallel subtraction and parallel division units implement the compression algorithm to compress the data from the original data into index values.

[0040] As an optional implementation, the data compression processor uses the DXT-5 algorithm for compression / decompression.

[0041] Reference Figure 2 As an optional implementation, the matrix unit includes a shift register group, a cross-group buffer circuit, and a timing control circuit. The cross-group buffer circuit includes multiple FIFO buffers. The timing control circuit is used to control the shift register group and the cross-group buffer circuit, so that the shift register group performs a shift operation or is set from its corresponding cross-group buffer circuit.

[0042] Specifically, the matrix unit comprises a shift register group, a cross-group buffer circuit, and a timing control circuit. In each independent clock cycle, the matrix unit is responsible for generating feature map data outputs for Np parallel convolutional windows. The timing control circuit controls the shift register group and cross-group buffer within the matrix unit, enabling the shift register group to perform shift operations or set bits from its corresponding cross-group buffer. The shift register group consists of shift registers with depth Nc, length Nx, and width Ny. The product of Nc, Nx, and Ny is the same as Np, i.e., Np = Nc * Nx * Ny. Here, the parallelism of Nc is for input feature map data multiplexing, and the parallelism of Nx and Ny is for weight data multiplexing. The cross-group buffer circuit consists of numerous FIFO buffers, which are responsible for enabling communication between the shift register groups to generate a timing-correct data stream.

[0043] Reference Figure 3 As an optional implementation, the convolutional unit includes multiple multiply-accumulate units. Each multiply-accumulate unit includes a control circuit, a dynamic shifter, a register, and a multiplier. The dynamic shifter is used to shift and align the input weight data and feature map data, and the register and multiplier are used to perform multiply-accumulate calculations on the shifted and aligned weight data and feature map data.

[0044] Specifically, a convolutional unit consists of multiple multiply-accumulate units, each including control circuitry, a dynamic shifter, registers, and multipliers. The dynamic shifter aligns the input weights and feature map data to accommodate the quantization bit width differences between different layers of the neural network. Registers and multipliers perform multiplication and partial summation of the input weights and feature map data. The control circuitry controls the remaining components.

[0045] As an optional implementation, the transfer engine includes a read DMA engine and a write DMA engine.

[0046] As an optional implementation, the logic control module includes an instruction cache unit, an instruction decoding unit, an input FSM, a convolutional FSM, and an output FSM.

[0047] As an optional implementation, the data caching module includes an input caching unit, an output caching unit, and a weight caching unit.

[0048] The above describes the structure and working principle of the digital converter according to embodiments of the present invention. It can be understood that embodiments of the present invention have the following advantages:

[0049] (1) The neural network accelerator architecture proposed in this invention benefits from a customized computational complex, which can improve the degree of data reuse within the accelerator, including weight reuse and data reuse, thereby achieving parallel acceleration of convolution computation; the parallelism is fully configurable and can adapt to convolutional layers of any size.

[0050] (2) The DXT-5 compression method for image compression was implemented on the hardware side and optimized to meet the behavior-driven burst transmission mode adopted by most hardware accelerators.

[0051] (3) An optimized computational complex with regular data flow was designed to meet the requirements of real-time image super-resolution tasks.

[0052] Reference Figure 5 This invention provides a method for implementing a neural network accelerator system for image super-resolution, which is used to implement the aforementioned neural network accelerator system for image super-resolution, and includes the following steps:

[0053] S101. The feature map data and weight data are transmitted to the data cache module through the transmission engine;

[0054] S102. Obtain control commands through the logic control unit and decode them, and then perform global control based on the decoding results;

[0055] S103. The feature map data stored in the data cache module is rearranged by the data routing unit and then transmitted to the data compression processor.

[0056] S104. The feature map data after data rearrangement is decompressed by the data compression processor and then transmitted to the matrix unit.

[0057] S105. Multiple parallel convolutional outputs are generated by the matrix unit based on the decompressed feature map data and transmitted to the convolution unit.

[0058] S106. The convolution unit performs multiplication and accumulation based on the parallel convolution output and the weight data stored in the data cache module to obtain the convolution calculation result and transmits it to the activation unit.

[0059] S107. The activation unit performs fitting processing on the convolution calculation result using a preset activation function to obtain the feature map data after convolution calculation and transmits it to the data compression processor.

[0060] S108. The feature map data after convolution calculation is compressed by the data compression processor and then transmitted to the data cache module.

[0061] It is understood that the content of the above system embodiments is applicable to this method embodiment. The specific functions implemented in this method embodiment are the same as those in the above system embodiments, and the beneficial effects achieved are also the same as those achieved in the above system embodiments.

[0062] It should be recognized that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium. The methods described above can be implemented using standard programming techniques—including implementation in a computer program on a non-transitory computer-readable storage medium configured to allow the computer to operate in a specific and predefined manner—according to the methods and drawings described in the specific embodiments. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. Furthermore, for this purpose, the program can run on a programmed application-specific integrated circuit (ASIC).

[0063] Furthermore, the procedures described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The procedures described herein (or variations and / or combinations thereof) may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The aforementioned computer programs include a plurality of instructions executable by one or more processors.

[0064] Furthermore, the above methods can be implemented in any suitable type of computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices, etc. Aspects of the invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it is readable by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein. Furthermore, the machine-readable code, or portions thereof, can be transmitted via wired or wireless networks. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media comprises instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor. When programmed according to the methods and techniques described in the invention, the invention also includes the computer itself.

[0065] A computer program can be applied to input data to perform the functions described herein, thereby transforming the input data to generate output data stored in non-volatile memory. The output information can also be applied to one or more output devices, such as a display. In a preferred embodiment of the invention, the transformed data represents physical and tangible objects, including a specific visual depiction of physical and tangible objects generated on the display.

[0066] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention, as long as they achieve the technical effects of the present invention by the same means, should be included within the scope of protection of the present invention. Within the scope of protection of the present invention, the technical solutions and / or implementation methods can have various modifications and variations.

Claims

1. A neural network accelerator system for image super-resolution, characterized in that: The system includes a transmission engine, a logic control module, a data cache module, a computational complex, an off-chip data bus, an off-chip control bus, an on-chip data bus, and an on-chip control bus. The transmission engine and the computational complex are both connected to the data cache module via the on-chip data bus. The transmission engine, the data cache module, and the computational complex are all connected to the logic control module via the on-chip control bus. The transmission engine is also connected to the off-chip data bus and is used for data transmission between the on-chip and off-chip caches. The logic control module is also connected to the off-chip control bus and is used for global control based on received control commands. The data cache module is used to cache feature map data and weight data. The computational complex is used to perform convolution computation tasks and includes a data routing unit, a data compression processor, a matrix unit, a convolution unit, and an activation unit, wherein: The data compression processor is used to compress / decompress the feature map data. The data routing unit is used to rearrange the feature map data stored in the data cache module and then transmit it to the data compression processor for decompression. The matrix unit is used to generate multiple parallel convolution outputs based on the decompressed feature map data. The convolution unit is used to multiply and accumulate the parallel convolution outputs and the weight data to obtain the convolution calculation result. The activation unit is used to fit the convolution calculation result with a preset activation function to obtain the convolution-calculated feature map data, and then compress it through the data compression processor before transmitting it to the data cache module for storage. The matrix unit includes a shift register group, a cross-group buffer circuit, and a timing control circuit. The cross-group buffer circuit includes multiple FIFO buffers. The timing control circuit is used to control the shift register group and the cross-group buffer circuit, so that the shift register group performs a shift operation or is set from its corresponding cross-group buffer circuit.

2. The neural network accelerator system for image super-resolution according to claim 1, characterized in that: The data compression processor includes a compression unit and a decompression unit. The compression unit includes a comparator tree, a first cache unit, a parallel subtraction unit, and a parallel division unit. The comparator tree is used to find the maximum and minimum values ​​in a compressed block and transmit them to the first cache unit. The parallel subtraction unit and the parallel division unit are used to perform compression processing based on the data cached by the first cache unit. The decompression unit includes a second cache unit, a subtraction unit, a parallel multiplication unit, and a parallel addition unit. The second cache unit is used to store the maximum and minimum values ​​in a compressed block. The subtraction unit, the parallel multiplication unit, and the parallel addition unit are used to perform decompression processing based on the data cached by the second cache unit.

3. The neural network accelerator system for image super-resolution according to claim 2, characterized in that: The data compression processor uses the DXT-5 algorithm for compression / decompression.

4. The neural network accelerator system for image super-resolution according to claim 1, characterized in that: The convolutional unit includes multiple multiply-accumulate units, each including a control circuit, a dynamic shifter, a register, and a multiplier. The dynamic shifter is used to shift and align the input weight data and feature map data, and the register and the multiplier are used to perform multiply-accumulate calculations on the shifted and aligned weight data and feature map data.

5. A neural network accelerator system for image super-resolution according to claim 1, characterized in that: The transfer engine includes a read DMA engine and a write DMA engine.

6. A neural network accelerator system for image super-resolution according to claim 1, characterized in that: The logic control module includes an instruction cache unit, an instruction decoding unit, an input FSM, a convolutional FSM, and an output FSM.

7. A neural network accelerator system for image super-resolution according to any one of claims 1 to 6, characterized in that: The data caching module includes an input caching unit, an output caching unit, and a weight caching unit.

8. A method for implementing a neural network accelerator system for image super-resolution, used by implementing the neural network accelerator system for image super-resolution as described in any one of claims 1 to 7, characterized in that, Includes the following steps: The feature map data and weight data are transmitted to the data cache module via the transmission engine. The logic control module acquires and decodes control commands, and then performs global control based on the decoding results. The feature map data stored in the data cache module is rearranged by the data routing unit and then transmitted to the data compression processor. The feature map data after data rearrangement is decompressed by the data compression processor and then transmitted to the matrix unit. The matrix unit generates multiple parallel convolutional outputs based on the decompressed feature map data and transmits them to the convolutional unit. The convolution unit performs multiplication and accumulation based on the parallel convolution output and the weight data stored in the data cache module to obtain the convolution calculation result, which is then transmitted to the activation unit. The activation unit performs fitting processing on the convolution calculation result using a preset activation function to obtain the convolutional feature map data and transmits it to the data compression processor. The feature map data after convolution calculation is compressed by the data compression processor and then transmitted to the data cache module.

Images