A method and device for accelerating AI computing

Abstract

The present invention belongs to the field of integrated circuit technology, and in particular, to a method and device for accelerating AI computing. It includes: step S1: the image signal processing unit receives the image data of the sensor, and determines whether an intervention signal is needed to intervene in the image data according to the configuration; step S2: judging the type of intervention mode configured; if the configuration type is intervention mode one, the pixel range of the content that needs to be retained is determined according to the threshold th and the constraint range dir; if the configuration type is intervention mode two, the image range of the content that needs to be retained is determined according to the regional mask reg_mask and the mask direction mask_dir; step S3: performing block-based image encoding on the pixel range or image range, and finally outputting the encoded data. The present invention solves the problem of redundant calculation in artificial intelligence calculation, and can effectively improve the performance of artificial intelligence calculation.

Description

Technical Field

[0001] The present invention belongs to the technical field of integrated circuits, and in particular, relates to a method and device for accelerating AI computing. Background Art

[0002] With the rapid development of science and technology, artificial intelligence is increasingly used in various fields, and higher and higher requirements are placed on the reasoning performance of chips and models. In traditional AI accelerated computing, a complete image data is collected by an image acquisition unit and input into the AI ​​reasoning model for calculation; however, in some application scenarios, the target information that is really needed in the field of view sometimes occupies a very small proportion of the image, while the background information accounts for a large proportion. That is, there are some areas that are not very useful in AI computing but have a relatively large proportion of data. When these redundant data are also transmitted and involved in AI computing, a large amount of unnecessary transmission, storage and computing resources will be consumed.

[0003] At present, there are two methods to accelerate the problem of redundant information. One is to perform sparse computing when the model is deployed to the AI ​​accelerator. However, this method requires the support of the AI ​​accelerator and cannot save the resources consumed in the data collection and transmission stages. Another method is to use solutions based on technologies such as compressed sensing when collecting information. However, the entire image needs to be completely reconstructed before AI reasoning, which still cannot reduce the amount of calculation. Moreover, the algorithms of these solutions are relatively fixed, and external control logic cannot intervene, which is not flexible enough. Summary of the invention

[0004] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art. The present invention provides a method and device for accelerating AI calculations, which utilizes the distribution characteristics of effective information for AI calculations in image signals, and an acceleration method that combines AI calculations with image acquisition and transmission, and adds external intervention to change the final output data volume of the unit; in order to be compatible with traditional image signal processing units at the same time, in the design of the AI ​​model, a post-feedback intervention method is added to change the overall calculation amount of the AI ​​model, and this method can act on both the AI ​​model itself and the image signal processing unit.

[0005] To solve the above technical problems, the present invention provides a method for accelerating AI computing, comprising the following steps:

[0006] Step S1: the image signal processing unit receives the image data of the sensor and determines whether an intervention signal is needed to intervene in the image data according to the configuration;

[0007] If the configuration needs to be intervened, then go to step S2; if the configuration does not need to be intervened, then compress the image data, and decide whether to compress and encode the image data before outputting it according to the configuration;

[0008] Step S2: Determine the configured intervention mode type;

[0009] First, the block ratio is determined by the control parameter blockdiv. If the configuration type is intervention mode 1, the pixel range of the content to be retained is determined based on the threshold th and the constraint range dir. If the configuration type is intervention mode 2, the image range of the content to be retained is determined based on the regional mask reg_mask and the mask direction mask_dir.

[0010] Step S3: Perform block-based image encoding on the pixel range or image range, and finally output the encoded data.

[0011] Preferably, in step S2, if the configuration type is intervention mode 1, the configured color mode includes but is not limited to YUV format and RGB format.

[0012] Preferably, in step S2, if the configuration type is intervention mode 1 and the color mode is YUV format, when the control parameters blockdiv=4, threshold th_y=10 and threshold th_rate=80%;

[0013] When the constraint range dir_y=1, it means that the obtained block size is 1 / 4 of the input image width and the shape is a square. When the proportion of pixels with color value Y<10 in the block reaches 80%, the image content of the block will be blocked.

[0014] When the constraint range dir_y=0, it means: the obtained block size is 1 / 4 of the input image width and the shape is a square. When the pixel proportion with color value Y>=10 in the block reaches 80%, the image content of the block will be blocked.

[0015] Preferably, in step S2, if the configuration type is intervention mode 1 and the color mode is YUV format, when the control parameters blockdiv=16, threshold th_y=21, threshold th_u=120, threshold th_v=145 and threshold th_rate=75%;

[0016] When the constraint range is dir_y=1, dir_u=0, and dir_v=0, the resulting block size is 1 / 16 of the input image width and the shape is a square. When the number of pixels in the block that meet the conditions Y<21, U>=120, and V>=145 reaches 75%, the image content of this block will be blocked.

[0017] Preferably, in step S2, if the configuration type is intervention mode 1 and the color mode is RGB format, when the control parameters blockdiv=8, threshold th_r=10, threshold th_g=15, threshold th_b=20 and threshold th_rate=70%;

[0018] When the constraint ranges dir_r, dir_g, and dir_b are all 1, the resulting block size is 1 / 8 of the input image width and the shape is a square. When the number of pixels in the block that meet the conditions of R<10, G<15, and B<20 reaches 70%, the image content of the block will be blocked.

[0019] When the constraint range dir_y is 0, when the number of pixels in the block that meet R>=10, G>=15, and B>=20 reaches 70%, the image content of this block will be blocked.

[0020] Preferably, in step S2, if the configuration type is intervention mode 2, when the control parameter blockdiv is 8 and the image size is 640×640, the image will be divided into 64 blocks and each block size is a square of 80×80, and the region mask reg_mask is composed of 64-bit binary numbers, which respectively represent blocks sorted in row priority from the upper left corner to the lower right corner. When the mask direction mask_dir is 1, the block image content with the region mask reg_mask being 1 will be shielded, and the block image content with the region mask reg_mask being 0 will be retained; when the mask direction mask_dir is 0, the block image content with the region mask reg_mask being 0 will be shielded, and the block image content with the region mask reg_mask being 1 will be retained.

[0021] Preferably, in step S1, an intervention signal can be generated during the AI ​​reasoning process by using an AI model; the AI ​​model establishes a branch network, namely an intervention signal network, between the neck and the head in the yolov5 model architecture, so that the head outputs an intervention signal that adds the regional mask reg_mask, threshold th and control parameter blockdiv content in the final output, while the intervention signal of the threshold th_rate, constraint range dir and mask direction mask_dir content is provided during model training in the form of hyperparameters; at the same time, the intervention signal output by the head is connected to the input end and the skeleton.

[0022] The present invention also provides a device for accelerating AI computing, which adopts a method for accelerating AI computing as described above, including an image signal processing unit, a coding transmission unit and an AI model.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention proposes a method and device that fully utilizes image features to accelerate AI computing in image acquisition, encoding transmission, AI computing, etc., and can flexibly use image features to reduce unnecessary transmission, storage and computing resource consumption. In a typical real-time use case, simulation results based on multiple image test sets show that the present invention can effectively reduce unnecessary resource consumption, and can save 23.6% to 78.2% of bandwidth resources and computing resources in different scenarios under test. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is an architectural diagram of the method and device of the present invention.

[0026] Figure 2 It is a processing flow chart of the image signal processing unit in the method of the present invention.

[0027] Figure 3 It is a schematic diagram of the AI ​​model structure in the method of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the accompanying drawings are in very simplified form and in non-precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0029] Figure 1Shown is an architecture diagram of the method and device of the present invention, which mainly controls image acquisition and its encoding process and AI reasoning process through intervention signals in traditional image acquisition, transmission and processing processes. The intervention signal can be generated by other means, or it can be generated during the AI ​​reasoning process to intervene in subsequent AI reasoning. There are two types of intervention signals. One is to determine the range of image content that does not need to be subsequently transmitted based on the threshold th and the constraint range dir (suitable for dynamic backgrounds). First, the block ratio is determined by the control parameter blockdiv; for example, parameter scheme 1: when the color mode is in YUV format, blockdiv is 4, th_y is 10, th_rate=80%, and dir_y is 1, the resulting block size is 1 / 4 of the width of the input image I, and the shape is a square. When the proportion of pixels with a color value less than 10 in the block in the specified color Y channel reaches 80%, the image content of this block will be blocked. When dir_y is 0, the proportion of pixels with a color value Y>=10 in the block reaches 80%, the image content of this block will be blocked; parameter scheme 2: when the color mode is in YUV format, blockdiv is 16, th_y=21, th_u=120, th_v=145, th_rate=75%, dir_ When y=1, dir_u=0, and dir_v=0, the resulting block size is 1 / 16 of the width of the input image I, and the shape is a square. When the number of pixels in the block that meet the conditions of Y<21, U>=120, and V>=145 reaches 75%, the image content of this block will be blocked. Parameter scheme 3: When the color mode is RGB format, blockdiv is 8, th_r=10, th_g=15, th_b=20, th_rate=70%, and dir_r, dir_g, and dir_b are all 1, the resulting block size is 1 / 8 of the width of the input image I, and the shape is a square. When the number of pixels in the block that meet the conditions of R<10, G<15, and B<20 reaches 70%, the image content of this block will be blocked. Similarly, when dir_y is 0, when the number of pixels in the block that meet the conditions of R>=10, G>=15, and B>=20 reaches 70%, the image content of this block will be blocked.

[0030] Another intervention signal is to determine the image content that does not need to be subsequently transmitted based on the regional mask reg_mask and the mask direction mask_dir (suitable for static background). First, the block ratio is determined by the control parameter blockdiv. For example, when blockdiv is 8 and the image size is 640×640, the image will be divided into 64 blocks and each block is 80×80 square. The regional mask consists of a 64-bit binary number, which represents the blocks sorted by row priority from the upper left corner to the lower right corner. When mask_dir is 1, the block image content with a mask value of 1 will be masked, and the block image content with a mask value of 0 will be retained. When mask_dir is 0, the masking situation is the opposite of when it is 1.

[0031] Figure 2 This is the processing flow of the image signal processing unit in the method of the present invention. The image processing unit first receives the image data of the sensor. If it is configured as no intervention is required, the traditional processing flow will be performed, and it will be determined whether the image needs to be compressed and encoded in a traditional way (such as JPEG compression) before output according to the configuration. For the case where the intervention is required, the processing logic implemented by the present invention will be entered. If it is configured as mode 1, the pixel range that does not need to retain the content is determined according to the th and dir values, and then the block-based image encoding is performed, and finally the encoded data is output; if it is configured as mode 2, the image range that does not need to retain the content is determined according to the reg_mask and mask_dir values, and then the block-based image encoding is performed, and finally the encoded data is output.

[0032] Figure 3 It is an example of the AI ​​model structure in the method of the present invention. Here, the general architecture of yolov5 is used as an example. The traditional yolov5 model is usually composed of several main parts: input end, skeleton, neck, and head. The skeleton is mainly used for feature extraction, the neck is mainly used for feature fusion, etc. to extract more advanced semantic information, and the head part further processes the features and outputs them. The present invention expands the functional structure similar to the neck and head parts in the AI ​​model, establishes a branch network-intervention signal network, and adds reg_mask, th, and blockdiv to the final output. For other parameters that need to be used (th_rate, dir, mask_dir), they are provided during model training by means of hyperparameters; therefore, the model expansion method can realize end-to-end training of the AI ​​model. After the model is trained, these intervention signals are connected to the input end and the skeleton part and fine-tuned to meet the requirements of being deployed in the AI ​​accelerator for reasoning.

[0033] In summary, the present invention makes full use of image features to accelerate the image acquisition, coding transmission, artificial intelligence AI calculation and other links. In the traditional image signal processing unit, a method of supporting external intervention is added to change the final output data volume of the unit; secondly, in the design of the artificial intelligence model, a post-feedback intervention method is added to change the overall calculation amount of the artificial intelligence model, and this method can act on the artificial intelligence model itself and the aforementioned image signal processing unit. The present invention solves the problem of redundant calculation in artificial intelligence calculation and can effectively improve the performance of artificial intelligence calculation.

[0034] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A method for accelerating AI computing, characterized in that: The steps include: Step S1: The image signal processing unit receives the image data of the sensor and determines whether an intervention signal is needed to intervene in the image data according to the configuration; If the configuration needs to be intervened, then go to step S2; if the configuration does not need to be intervened, then compress the image data, and decide whether to compress and encode the image data before outputting it according to the configuration; Step S2: Determine the configured intervention mode type; First, the block ratio is determined by the control parameter blockdiv. If the configuration type is intervention mode 1, the pixel range of the content to be retained is determined based on the threshold th and the constraint range dir. If the configuration type is intervention mode 2, the image range of the content to be retained is determined based on the regional mask reg_mask and the mask direction mask_dir. Step S3: performing block-based image encoding on the pixel range or the image range, and finally outputting encoded data; In the step S1, an intervention signal can be generated in the AI ​​reasoning process by using an AI model; the AI ​​model establishes a branch network, namely an intervention signal network, between the neck and the head in the yolov5 model architecture, so that the head outputs an intervention signal with the added content of the regional mask reg_mask, the threshold th and the control parameter blockdiv in the final output, while the intervention signal of the content of the threshold th_rate, the constraint range dir and the mask direction mask_dir is provided during the model training in a hyperparameter manner; at the same time, the intervention signal output by the head is connected to the input end and the skeleton; In step S2, if the configuration type is intervention mode 1, the configured color mode includes YUV format and RGB format; In step S2, if the configuration type is intervention mode 1 and the color mode is YUV format, when the control parameters blockdiv=4, threshold th_y=10 and threshold th_rate=80%; When the constraint range dir_y=1, it means that the obtained block size is 1 / 4 of the input image width and the shape is a square. When the proportion of pixels with color value Y<10 in the block reaches 80%, the image content of the block will be blocked. When the constraint range dir_y=0, it means: the obtained block size is 1 / 4 of the input image width and the shape is a square. When the pixel proportion of color value Y>=10 in the block reaches 80%, the image content of the block will be blocked; In step S2, if the configuration type is intervention mode 1 and the color mode is YUV format, when the control parameters blockdiv=16, threshold th_y=21, threshold th_u=120, threshold th_v=145 and threshold th_rate=75%; When the constraint range is dir_y=1, dir_u=0, and dir_v=0, the resulting block size is 1 / 16 of the input image width and the shape is a square. When the number of pixels in the block that meet the conditions of Y<21, U>=120, and V>=145 reaches 75%, the image content of the block will be blocked. In step S2, if the configuration type is intervention mode 1 and the color mode is RGB format, when the control parameters blockdiv=8, threshold th_r=10, threshold th_g=15, threshold th_b=20 and threshold th_rate=70%; When the constraint ranges dir_r, dir_g, and dir_b are all 1, the resulting block size is 1 / 8 of the input image width and the shape is a square. When the number of pixels in the block that meet the conditions of R<10, G<15, and B<20 reaches 70%, the image content of the block will be blocked. When the constraint range dir_y is 0, if the number of pixels in the block that meet the conditions of R>=10, G>=15, and B>=20 reaches 70%, the image content of the block will be blocked; In step S2, if the configuration type is intervention mode 2, when the control parameter blockdiv is 8 and the image size is 640×640, the image will be divided into 64 blocks and each block size is a square of 80×80. The region mask reg_mask is composed of 64-bit binary numbers, which represent blocks sorted in row priority from the upper left corner to the lower right corner. When the mask direction mask_dir is 1, the image content of the block with the region mask reg_mask of 1 will be masked, and the image content of the block with the region mask reg_mask of 0 will be retained. When the mask direction mask_dir is 0, the block image content where the region mask reg_mask is 0 will be masked, and the block image content where the region mask reg_mask is 1 will be retained.

2. A device for accelerating AI calculation, using the method for accelerating AI calculation as claimed in claim 1, characterized in that: Includes image signal processing unit, encoding transmission unit and AI model.

Images