A novel accelerated configurable image convolution data conversion system and method

By designing a new accelerated image convolution data conversion system in deep learning, using convolution arrays for weight preprocessing and data remapping, the problem of inefficient image convolution operation calculation is solved, and efficient image convolution acceleration and resources are achieved use.

CN118628335BActive Publication Date: 2024-11-19CLP KESHENTAI INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411118737.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-15
Publication Date
2024-11-19
Estimated Expiration
2044-08-15

AI Technical Summary

Technical Problem

The prior art has inefficient computing efficiency and waste of resources in deep learning, and cannot fully utilize the parallel computing power of hardware, especially when processing large-scale image data sets.

Method used

A new type of accelerated configurable image convolution data conversion system is designed, including image convolution weight preprocessing module, image pixel operand generation module, image data request single module, image data response processing module, image data accuracy conversion module and volume The product sequence generator makes full use of convolutional arrays for convolutional operations through weight preprocessing, data remapping, filling and precision conversion.

Benefits of technology

It improves the computational parallelism, increases the computing efficiency of convolutional networks in deep learning, effectively utilizes hardware resources, is suitable for different pixel spaces and hardware platforms, and realizes efficient acceleration of image convolution operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118628335B_ABST
    Figure CN118628335B_ABST
Patent Text Reader

Abstract

The present invention relates to a novel accelerated configurable image convolution data conversion system and method, wherein the data conversion system comprises an image convolution weight preprocessing module, an image pixel operand generation module, an image data request single module, an image data response processing module, an image data precision conversion module, a convolution weight data cache unit, and a convolution sequence generator, wherein the image data precision conversion module receives data from the image data response processing module, and performs quantization and precision conversion operations on the data according to configuration information, realizes precision conversion of image convolution data, and stores the convolution data in the convolution weight data cache unit. The system and method make full use of convolution arrays and parallel computing acceleration based on image convolution preprocessing weights, and has flexible and configurable data conversion matching hardware information, which saves the storage resources of the NPU while realizing the acceleration of convolution in the channel C direction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit and deep learning acceleration unit design, and in particular to a novel accelerated configurable image convolution data conversion system and method. Background Art

[0002] Image convolution is one of the commonly used operations in deep learning to extract features from images. In convolutional neural networks (CNNs), convolution operations are usually computationally intensive, especially for large-scale image datasets.

[0003] As network models continue to grow in size and complexity, conventional acceleration methods have gradually become less efficient. Conventional neural network hardware acceleration is often limited by the computing power of the CPU or GPU and cannot fully tap the potential of the hardware. In addition, conventional hardware design methods often result in a waste of computing resources and cannot effectively utilize the parallel computing capabilities of the hardware, resulting in low computing efficiency and waste of computing resources. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a novel accelerated configurable image convolution data conversion system, which is designed based on an integrated circuit for a deep learning acceleration unit of an image, and includes an image convolution weight preprocessing module, an image pixel operand generation module, an image data request unit module, an image data response processing module, an image data precision conversion module, a convolution weight data cache unit, and a convolution sequence generator;

[0005] The image convolution weight preprocessing module receives the image convolution mode from the configuration bus, as well as the number, width, height, channel of the input weight convolution kernel and the basic address information of the weight stored in the memory, and generates a weight read request based on this, and preprocesses the weight of the response according to the array information available for use of the convolution array, and stores the preprocessed weight in the convolution weight data cache unit according to the configuration information;

[0006] The image pixel operand generation module receives the image pixel configuration information from the configuration bus, generates pixel operand information according to the pixel information, and the operand information includes the plane information flag, precision information, pixel reordering information, 10b pixel information flag, plane 0 pixel positioning, plane 1 pixel positioning, plane 0 pixel mask, and plane 1 pixel mask of the image pixel, and provides operation support for subsequent pixel remapping and filling of different image data;

[0007] The image data request module receives the width, height and channel information of the cube of the image data from the configuration bus, and receives the pixel operand information from the image pixel operand generation unit, and generates an image data read request according to the information of the cube of the input image, the basic address of the data cube stored in the memory and the pixel information;

[0008] The image data response processing module reorders the pixel data according to the image pixel format and the corresponding generated pixel operands, and fills the left and right padding of the configuration of the image convolution input cube;

[0009] The image data precision conversion module receives data from the image data response processing module, and performs quantization and precision conversion operations on the data according to the configuration information, realizes the precision conversion of the image convolution data, and stores the convolution data into the convolution weight data cache unit;

[0010] The convolution sequence generator, based on the method of image convolution preprocessing weights, makes full use of the convolution array and sends weights and data to the convolution array for convolution operation, thereby achieving the purpose of accelerating the convolution in the channel C direction.

[0011] In one embodiment of the present invention, the image convolution weight preprocessing module has the functions of sending read requests for memory data, preprocessing the weights of the responses, and reading and writing the convolution weight data cache unit; when the bus is configured to configure the convolution mode as the image convolution mode, and the number, width, height and channel information of the weight convolution kernels are input, a weight read request is generated based on this, the read request information includes the request address, the requested burst data length, the weights waiting for the response from the memory, matching the weight read request packet, and storing the weight information in the convolution weight data buffer unit, and reading the weights from the convolution weight data buffer unit based on the array information available for use of the hardware convolution array, and preprocessing the weights. The purpose of preprocessing the weights is to make full use of the multiplication units in the convolution array, and the preprocessed weights are written to the convolution weight data buffer unit, leaving them for use by the convolution sequence generator.

[0012] In one embodiment of the present invention, in the image data response processing module, when the image data response processing module receives the response data, it matches the image data request packet, writes the image data into the convolution weight data buffer sequence, reorders the pixel data according to the image pixel format and the corresponding generated pixel operands, and fills the left and right padding of the configuration of the image convolution input cube; and then sends the processed data to the image data precision conversion module.

[0013] In one embodiment of the present invention, the image data precision conversion module receives data from the image data response processing module, and performs quantization and precision conversion operations on the data according to the configuration information. The quantization formula is out=(data-offset)*scale>>shift, wherein data is input data, offset is offset, scale and shift constitute scaling factors, and implement the image convolution data quantization step. The precision conversion is to saturate and truncate the data to prevent the processed data from overflowing, and the quantized and precision converted data is stored in the convolution weight data cache unit.

[0014] The present invention also provides a novel accelerated configurable image convolution data conversion method. According to the data conversion system construction design, the data conversion method includes weight preprocessing and image data processing at the same time, and is parallel processing; wherein part of the weight preprocessing method includes the following steps:

[0015] Step S1: Read in weight configuration information, including the number, width, height, and channels of weighted convolution kernels (K*R*S*C);

[0016] Step S2: store the response weights into the convolution weight data cache unit according to the priority of c>K>R>S>C, where c is the size of the response weight read from the memory, K is the number of convolution kernels, R is the width of the convolution kernel, S is the height of the convolution kernel, and C is the number of channels of the convolution kernel;

[0017] Step S3: remap the weights read from the convolution weight data cache unit in the order of K*1*S*(C+R*C);

[0018] Step S4: According to the multiplier number information Cm of the convolution array, determine whether the preprocessed weights can fully utilize the multipliers of the convolution array. If not, rearrange the weight convolution kernels to be arranged into K*1*S / N*(C+N*R*C), where N is the height at which the weight convolution kernel is divided;

[0019] Step S5: Buffer the final preprocessed weights into the convolution weight data cache unit.

[0020] In one embodiment of the present invention, part of the method for image data processing includes the following steps:

[0021] Step SⅠ: read in data configuration information, including the width, height, and channels (W*H*C) of the input image data cube;

[0022] Step SⅡ: store the response image data into the convolution weight data cache unit according to the priority of C>W>H>c, where c is the size of the response data read from the memory, W is the width of the response data cube, H is the height of the response data cube, and C is the number of channels of the response data cube;

[0023] Step SⅢ: the image data read out from the convolution weight data cache unit is remapped to different types of image pixels based on the pixel operand information from the image pixel operand generation module. If the pixel space is RGB space, the pixels are remapped into RGB. If the pixel space is YUV, the pixels are remapped into YUV format regardless of whether they are semi-plane or single-plane.

[0024] Step SIV: inserting left and right padding values ​​into the remapped image pixel data of the image pixel according to the padding configuration information and the pixel operand information;

[0025] Step SⅤ: Finally, the filled pixel data is passed through the image data precision conversion module, and the pixel data is written into the convolution weight data cache unit.

[0026] The above technical solution of the present invention has the following advantages compared with the prior art:

[0027] (1) Flexible support for different pixel spaces RGB or YUV. It can be configured to make full use of the calculation cache for different pixel spaces;

[0028] (2) A data conversion system and method, which has flexible and configurable data conversion that matches hardware information, increases the computational parallelism of convolutional networks in deep learning, and achieves the effect of accelerating deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings.

[0030] Figure 1 It is the overall block diagram of a novel configurable accelerated image convolution data conversion system in NPU proposed by the present invention;

[0031] Figure 2 It is the overall block diagram of the data conversion system proposed by the present invention;

[0032] Figure 3 It is a schematic block diagram of weight processing by the weight preprocessing module of the present invention;

[0033] Figure 4 It is a schematic block diagram of pixel data processing by the image data response module of the present invention. DETAILED DESCRIPTION

[0034] like Figure 1 As shown, this embodiment provides a new configurable accelerated image convolution data conversion system in the deep learning acceleration unit, which is in the weight and data preparation stage of the convolution operation. Its main purpose is to realize the remapping of the convolution weights and image data, so as to achieve the acceleration of the convolution network operation.

[0035] The RTL code circuit implementation based on NPU hardware is as follows Figure 2 A new type of configurable accelerated image convolution data conversion system with weight and data remapping and data filling functions, including image convolution weight preprocessing module, image pixel operand generation module, image data request single module, image data response processing module, image data precision conversion module, convolution weight data cache unit, and convolution sequence generator.

[0036] Figure 2 The image convolution weight preprocessing module implements the functions of sending read requests for memory data, preprocessing the weights of the responses, and reading and writing the convolution weight data cache unit. When the bus is configured to configure the convolution mode as the image convolution mode, and the number, width, height, and channel (K*R*S*C) information of the input weight convolution kernels, a weight read request is generated based on this. The read request information contains the request address, the length of the requested burst data, and waits for the weight data from the memory response to match the weight read request packet, and stores the weight information in the convolution weight data buffer unit. According to the array information available for use by the hardware convolution array, the weights are read from the convolution weight data buffer unit and preprocessed. The purpose of preprocessing the weights is to make full use of the multiplication units in the convolution array, and the preprocessed weights are written to the convolution weight data buffer unit, leaving them for the convolution sequence generator to use.

[0037] Figure 2 The image pixel operand generation module generates pixel operand generation functions. It receives the image pixel configuration information from the configuration bus and generates pixel operand information according to the pixel information. The operand information includes the plane information flag, precision information, pixel reordering information, and 10b pixel information flag, plane 0 pixel positioning, plane 1 pixel positioning, plane 0 pixel mask, and plane 1 pixel mask of the image pixel, providing operation support for subsequent pixel remapping and filling of different image data.

[0038] Figure 2The image data request module sends a read request to the memory data read. When the image data request module receives the width, height and channel (W*H*C) information of the image data cube from the configuration bus, and receives the pixel operand information from the image pixel operand generation unit, it generates an image data read request based on the information of the input image cube, the basic address of the data cube stored in the memory, and the pixel information. The read request information includes the request address, the burst data length of the request, and whether the request is an empty request.

[0039] Figure 2 The image data response processing module implements the functions of receiving image data, reading and writing convolution weight data cache units, rearranging image data according to pixel patterns, and padding the input image data cube on the left and right. When the image data response processing module receives the response data, it matches the image data request packet, writes the image data into the convolution weight data buffer sequence, reorders the pixel data according to the image pixel format and the corresponding generated pixel operands, and pads the left and right padding of the image convolution input cube. The processed data is then sent to the image data precision conversion module.

[0040] Figure 2 The image data precision conversion module implements the functions of data precision conversion and configurable quantization, accepts data from the image data response processing module, and performs quantization and precision conversion operations on the data according to the configuration information. The quantization formula is out=(data-offset)*scale>>shift, where data is the input data, offset is the offset, scale and shift constitute the scaling factor, and the image convolution data quantization step is implemented. The precision conversion is to saturate and truncate the data to prevent the processed data from overflowing, and the quantized and precision converted data is stored in the convolution weight data cache unit.

[0041] Figure 2 The function of generating a convolution sequence by the convolution sequence generator is implemented in . According to the method of preprocessing weights of image convolution, the convolution array is fully utilized to read the weights and data from the convolution weight data cache unit to generate an efficient convolution sequence. The weights and data are sent to the convolution array for convolution operation, thereby achieving the purpose of accelerating the convolution in the channel C direction.

[0042] In addition, if Figure 3 , which is a schematic diagram of remapping of a 4*4*4 (R*S*C) convolution kernel weight. The image convolution weight preprocessing module remaps the weights according to the different hardware resources of the NPU convolution array. The method steps are as follows:

[0043] S1: Read in weight configuration information, including the number, width, height, and channels of weight convolution kernels;

[0044] S2: Store the response weights into the convolution weight data cache unit according to the priority of c>K>R>S>C, where c is the size of the response weight read from the memory;

[0045] S3: remap the weights read from the convolution weight data cache unit in the order of K*1*S*(C+R*C);

[0046] S4: According to the information Cm of the number of multipliers of the convolution array, determine whether the preprocessed weights can fully utilize the multipliers of the convolution array. If not, rearrange the weight convolution kernels to be arranged into K*1*S / N*(C+N*R*C), where N is the height at which the weight convolution kernels are divided;

[0047] S5: Write the final preprocessed weights into the convolution weight data cache unit.

[0048] Likewise, if Figure 4 , which shows a schematic diagram of the remapping and processing of an 8*128*4 (W*H*C) image. The image pixels also have a similar remapping and processing method in the image data to be convolved. The method and steps are as follows:

[0049] SⅠ: Read in data configuration information, including the width, height, and channels of the input image data cube;

[0050] SⅡ: Store the response image data into the convolution weight data cache unit according to the priority of C>W>H>c, where c is the size of the response data read from the memory;

[0051] SⅢ: The image data read from the convolution weight data cache unit is remapped to different types of image pixels based on the pixel operand information from the image pixel operand generation module. If the pixel space is RGB space, the pixels are remapped to RGB. If the pixel space is YUV, whether it is half-plane or single-plane pixels, they are remapped to YUV format.

[0052] SⅣ: insert the left and right padding values ​​into the remapped image pixel data of the image pixel according to the padding configuration information and the pixel operand information.

[0053] SⅤ: Finally, the padded pixel data is passed through the image data precision conversion module and the pixel data is written into the convolution weight data cache unit.

[0054] In order to speed up the calculation process of the convolution operation, a specially designed hardware accelerator is used to accelerate the calculation of the convolutional neural network. Conventional chips used for hardware acceleration of convolutional neural networks include GPU, FPGA, NPU, TPU, etc. In addition, the convolutional neural network can also be pruned. The network parameters and the amount of calculation can be reduced through network pruning technology, thereby accelerating the calculation of the convolutional neural network. The convolutional neural network can also be quantized: quantizing the network parameters to a low number of bits can reduce the computational complexity and accelerate the calculation speed of the network. Similarly, data reuse in the convolutional neural network can reduce data transmission and storage overhead and accelerate the calculation speed of the convolutional neural network. In addition to the above methods, parallel computing technology can also be used to accelerate the calculation of the convolutional neural network.

[0055] In this embodiment, based on the hardware of the neural network acceleration unit NPU, a new accelerated configurable image convolution data conversion system and method is proposed using parallel computing technology. The system can automatically adjust the data layout according to the compiler based on the hardware platform and network structure, and make full use of the storage and computing resources of the hardware to achieve more efficient image convolution operations. The main advantages of this technology include high flexibility, strong adaptability, good versatility, etc., and it can achieve efficient image convolution operations on different network structures.

[0056] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the protection scope of the invention.

Claims

1. A novel accelerated configurable image convolution data conversion system, the data conversion system is designed based on the deep learning acceleration unit of the integrated circuit for the image, characterized in that: It includes an image convolution weight preprocessing module, an image pixel operand generation module, an image data request unit module, an image data response processing module, an image data precision conversion module, a convolution weight data cache unit, and a convolution sequence generator; The image convolution weight preprocessing module receives the image convolution mode from the configuration bus, as well as the number, width, height, channel of the input weight convolution kernel and the basic address information of the weight stored in the memory, and generates a weight read request based on this, and preprocesses the weight of the response according to the array information available for use of the convolution array, and stores the preprocessed weight in the convolution weight data cache unit according to the configuration information; The image pixel operand generation module receives the image pixel configuration information from the configuration bus, generates pixel operand information according to the pixel information, and the operand information includes the plane information flag, precision information, pixel reordering information, 10b pixel information flag, plane 0 pixel positioning, plane 1 pixel positioning, plane 0 pixel mask, and plane 1 pixel mask of the image pixel, and provides operation support for subsequent pixel remapping and filling of different image data; The image data request module receives the width, height and channel information of the cube of the image data from the configuration bus, and receives the pixel operand information from the image pixel operand generation unit, and generates an image data read request according to the information of the cube of the input image, the basic address of the data cube stored in the memory and the pixel information; The image data response processing module reorders the pixel data according to the image pixel format and the corresponding generated pixel operands, and fills the left and right padding of the configuration of the image convolution input cube; The image data precision conversion module receives data from the image data response processing module, and performs quantization and precision conversion operations on the data according to the configuration information, realizes the precision conversion of the image convolution data, and stores the convolution data into the convolution weight data cache unit; The convolution sequence generator, based on the method of image convolution preprocessing weights, makes full use of the convolution array and sends weights and data to the convolution array for convolution operation, thereby achieving the purpose of accelerating the convolution in the channel C direction.

2. The novel accelerated configurable image convolution data conversion system according to claim 1, characterized in that: The image convolution weight preprocessing module has the functions of sending read requests and responding to memory data, preprocessing weights, and reading and writing convolution weight data cache units; when the bus is configured to configure the convolution mode as the image convolution mode, and the number, width, height, and channel information of the weight convolution kernels are input, a weight read request is generated based on this, the read request information includes the request address, the requested burst data length, and the weights waiting for the response from the memory are matched with the weight read request packet, and the weight information is stored in the convolution weight data buffer unit, and the weights are read from the convolution weight data buffer unit based on the array information available for use of the hardware convolution array, and the weights are preprocessed. The purpose of preprocessing the weights is to make full use of the multiplication units in the convolution array, and the preprocessed weights are written into the convolution weight data buffer unit, leaving them for use by the convolution sequence generator.

3. The novel accelerated configurable image convolution data conversion system according to claim 1, characterized in that: In the image data response processing module, when the image data response processing module receives the response data, it matches the image data request packet and writes the image data into the convolution weight data buffer sequence.

4. The novel accelerated configurable image convolution data conversion system according to claim 1, characterized in that: The image data precision conversion module receives data from the image data response processing module, and performs quantization and precision conversion operations on the data according to the configuration information. The quantization formula is out=(data-offset)*scale>>shift, where data is input data, offset is offset, scale and shift constitute scaling factors, and implement the image convolution data quantization step. The precision conversion is to saturate and truncate the data to prevent the processed data from overflowing, and the quantized and precision converted data is stored in the convolution weight data cache unit.

5. A novel accelerated configurable image convolution data conversion method, according to the data conversion system construction design described in any one of claims 1-4, characterized in that: The data conversion method also includes weight preprocessing and image data processing; wherein part of the weight preprocessing method is as follows: Step S1: Read in weight configuration information, including the number, width, height, and channels of weight convolution kernels; Step S2: store the response weights into the convolution weight data cache unit according to the priority of c>K>R>S>C, where c is the size of the response weight read from the memory, K is the number of convolution kernels, R is the width of the convolution kernel, S is the height of the convolution kernel, and C is the number of channels of the convolution kernel; Step S3: remap the weights read from the convolution weight data cache unit in the order of K*1*S*(C+R*C); Step S4: According to the multiplier number information Cm of the convolution array, determine whether the preprocessed weights can fully utilize the multipliers of the convolution array. If not, rearrange the weight convolution kernels to be arranged into K*1*S / N*(C+N*R*C), where N is the height at which the weight convolution kernel is divided; Step S5: writing the final preprocessed weight into the convolution weight data cache unit; Some methods of image data processing are as follows: Step SⅠ: read in data configuration information, including the width, height, and channels of the input image data cube; Step SⅡ: store the response image data into the convolution weight data cache unit according to the priority of C>W>H>c, where c is the size of the response data read from the memory, W is the width of the response data cube, H is the height of the response data cube, and C is the number of channels of the response data cube; Step SⅢ: the image data read out from the convolution weight data cache unit is remapped to different types of image pixels based on the pixel operand information from the image pixel operand generation module. If the pixel space is RGB space, the pixels are remapped into RGB. If the pixel space is YUV, the pixels are remapped into YUV format regardless of whether they are semi-plane or single-plane. Step SIV: inserting left and right padding values ​​into the remapped image pixel data of the image pixel according to the padding configuration information and the pixel operand information; Step SⅤ: Finally, the filled pixel data is passed through the image data precision conversion module, and the pixel data is written into the convolution weight data cache unit.