Inference acceleration method and device of embedded edge intelligent computing system

Abstract

The invention provides a reasoning acceleration method and device for an embedded edge intelligent computing system and a storage medium, and the method comprises the steps: integrating a CPU (Central Processing Unit) and an NPU (Network Processing Unit) processor, and constructing a basic heterogeneous architecture of edge computing; configuring a model offline deployment framework, wherein the framework supports deployment of the trained YOLO model on an NPU processor; performing logic reconstruction on a detection layer operator network structure of the YOLO model; and operator programming and multi-core acceleration calculation are carried out on the reconstructed YOLO model detection layer. According to the method, the operator speed is increased, and the system reasoning speed is increased.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing equipment and relates to an inference acceleration method, device and storage medium for an embedded edge intelligent computing system. Background Art

[0002] Object detection algorithm is one of the core tasks in the field of computer vision. Its main research goal is to achieve accurate recognition and precise positioning of specific categories of targets in static images or dynamic video data. As a model of single-stage object detection algorithm, the YOLO series of algorithms have been widely recognized and applied in academia and industry for their efficient detection speed and accurate recognition capabilities.

[0003] There are two main ways to deploy deep learning models on edge devices. One is to deploy the deep learning model in a remote cloud data center. The mobile device transmits the input data to the cloud. After the cloud completes the inference task, the result is transmitted back to the mobile device. However, data transmission to the cloud will cause high latency, round-trip delay, security and privacy issues, and inability to make real-time decisions. The second is to deploy the model on the edge side and perform inference on the edge side. In order to meet the computing power and real-time requirements of the edge object detection system, a CPU+NPU heterogeneous intelligent computing system is usually built to provide higher computing power.

[0004] However, with the continuous evolution of deep learning technology, the architecture of the YOLO model has become more complex and sophisticated, and the number of model parameters has increased dramatically. Although this trend has made it possible to improve algorithm performance, it has also put forward higher requirements on computing resources. At the same time, the model is difficult to deploy, operators are not supported, and the reasoning speed is slow and cannot meet the real-time requirements. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes an inference acceleration method for an embedded edge intelligent computing system, an inference acceleration device for an embedded edge intelligent computing system, and a storage medium.

[0006] In order to achieve the above object, the present invention provides, on one hand, a method for accelerating reasoning of an embedded edge intelligent computing system, comprising:

[0007] Integrate CPU and NPU processors to build a basic heterogeneous architecture for edge computing;

[0008] Configure the model offline deployment framework, which supports deploying the trained YOLO model on the NPU processor;

[0009] Logically reconstructing the detection layer operator network structure of the YOLO model;

[0010] Perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

[0011] In some embodiments, the model offline deployment framework is configured with:

[0012] NPU inference thread: used to send the preprocessed image data to the NPU inference network for inference, where the NPU inference network only performs inference to the layer before the detection layer of the YOLO model to obtain the NPU inference result.

[0013] In some embodiments, the detection layer operator network structure of the YOLO model is logically reconstructed, including:

[0014] Use the CPU inference network to infer the detection layer operators of the YOLO model and perform bounding box prediction and category prediction.

[0015] In some embodiments, before performing border prediction and category prediction, the method further includes:

[0016] Using the NPU inference result as first input data, and splitting the first input data into first sub-data and second sub-data according to a first preset dimension;

[0017] Wherein: the dimension of the first sub-data is greater than the dimension of the second sub-data, and the first sub-data is used for border prediction, and the second sub-data is used for category prediction.

[0018] In some embodiments, the border prediction process includes:

[0019] Perform dimension upgrading and transposition on the first sub-data to obtain second data;

[0020] Perform a softmax operation on the second data to obtain third data;

[0021] A convolution operation is performed on the third data to obtain a final border prediction result.

[0022] In some embodiments, before the softmax operation, the second data is pre-dimensionalized, and the dimensionally reduced data is transposed to obtain fourth data.

[0023] In some embodiments, the process of class prediction includes:

[0024] The second sub-data is input into a prediction model based on the sigmoid function, and a final category prediction result is output.

[0025] In some embodiments, in single-core and multi-core cases, BANGC operator programming and SIMD multi-core accelerated computing are performed on the reconstructed YOLO model detection layer.

[0026] In some embodiments, in a single-core case, a collect mask collector is used to rearrange the fourth data, and an addition operation is performed on the third data instead of a convolution operation.

[0027] In some embodiments, in a multi-core case, the fourth data is decomposed into multiple grouped data and distributed to each core in the multi-core, and each core executes: using a collect mask collector to rearrange the grouped data corresponding to the core, and using an addition operation instead of a convolution operation.

[0028] In some embodiments, in a multi-core scenario, an inter-core shared memory SRAM is used instead of a global memory GDRAM.

[0029] In some embodiments, the YOLO model adopts YOLOv8.

[0030] Another aspect of the present invention further provides an inference acceleration device for an embedded edge intelligent computing system, comprising:

[0031] A framework configuration module to integrate CPU and NPU processors to build a basic heterogeneous architecture for edge computing; and a model offline deployment framework that supports deploying the trained YOLO model on the NPU processor;

[0032] A network optimization module, used to logically reconstruct the detection layer operator network structure of the YOLO model;

[0033] The acceleration configuration module is used to perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

[0034] Another aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned inference acceleration method of the embedded edge intelligent computing system.

[0035] It can be seen from the above scheme that the advantages of the present invention are:

[0036] The present invention discloses an inference acceleration method for an embedded edge intelligent computing system, which integrates CPU and NPU processors, builds a basic heterogeneous architecture for edge computing, and configures a model offline deployment framework, which supports the deployment of the trained YOLO model on the NPU processor. Then, the operator network structure of the detection layer of the YOLO model is logically reconstructed, and operator programming and multi-core acceleration calculation are performed on the reconstructed YOLO model detection layer. This method improves the operator speed and the system reasoning speed. This method realizes the construction of an embedded intelligent computing system with high computing power and strong real-time performance that meets edge-side deployment applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 The overall flow diagram of the inference acceleration method of the embedded edge intelligent computing system provided by one embodiment of the present invention is shown;

[0038] Figure 2 The overall data flow diagram of the logical reconstruction of the detection layer operator network structure of the YOLO model is shown;

[0039] Figure 3 The data structure flow chart of the DFL operator before optimization is shown; Figure 4 The optimized DFL operator data structure flow chart is shown;

[0040] Figure 5 The MLU memory space structure diagram of the deep learning processor is shown;

[0041] Figure 6 The flowchart of the DFL operator using the collect mask collector for merging and summing is shown;

[0042] Figure 7 A schematic diagram of the structure of an inference acceleration device for an embedded edge intelligent computing system is shown;

[0043] in:

[0044] 200: an inference acceleration device for an embedded edge intelligent computing system;

[0045] 210: framework configuration module;

[0046] 220: network optimization module;

[0047] 230: Acceleration configuration module. DETAILED DESCRIPTION

[0048] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0049] See also Figure 1-Figure 6 , Figure 1 The overall flow diagram of the inference acceleration method of the embedded edge intelligent computing system provided by one embodiment of the present invention is shown; Figure 2 The overall data flow diagram of the logical reconstruction of the detection layer operator network structure of the YOLO model is shown; Figure 3 The data structure flow chart of the DFL operator before optimization is shown; Figure 4 The optimized DFL operator data structure flow chart is shown; Figure 5 The MLU memory space structure diagram of the deep learning processor is shown; Figure 6 The flowchart of the DFL operator using the collect mask collector for merge and summation is shown.

[0050] See also Figure 1 As shown in , the reasoning acceleration method of the embedded edge intelligent computing system specifically includes the following steps:

[0051] Step S1: Integrate CPU and NPU processors to build a basic heterogeneous architecture for edge computing.

[0052] In some embodiments, a domestic CPU processor + a domestic NPU processor are used as a basis to construct an embedded heterogeneous intelligent computing architecture, but the present invention is not limited to this.

[0053] Step S2: Configure a model offline deployment framework, which supports deploying the trained YOLO model on an NPU processor.

[0054] The configuration of the YOLO model offline deployment framework mainly uses C++ to rewrite the model so that the model can be decoupled from the Pytorch framework for offline reasoning. The framework supports deploying the trained YOLO model on the NPU processor. Specifically, the trained YOLO model can be deployed on the deep learning processor MLU.

[0055] The configuration process mainly includes: NPU inference thread: It is used to send the preprocessed image data to the NPU inference network for inference. At this stage, since the NPU cannot fully support the YOLO detection layer operator, the NPU inference network inference only proceeds to the layer before the detection layer of the YOLO model to obtain the NPU inference result.

[0056] The YOLO detection layer operator function is implemented by the CPU alone. The CPU inference network performs bounding box prediction and category prediction on the YOLO model detection layer to achieve logical reconstruction of the detection layer operator network structure of the YOLO model. This process will be described in detail in step S3 and will not be described in detail here.

[0057] In addition, before executing the NPU inference thread, the YOLO model offline deployment framework is also configured with a data preparation preprocessing thread, which is used to complete the preliminary data preparation, including binding the device ID, allocating memory for the input data, converting the original image data into preprocessed image data and storing the results.

[0058] In addition, after executing the NPU inference thread, the YOLO model offline deployment framework is also configured with a post-processing thread, which receives the inference results of the layers outside the YOLO model detection layer from the NPU for post-processing on the one hand, and receives the inference results of the YOLO model detection layer from the CPU for post-processing operations on the other hand, to implement functions such as binding the device ID and setting the output data format, processing the output data and calculating the final bounding box, processing and visualizing the output data, etc.

[0059] In some embodiments, the YOLO model uses YOLOv8, which is a deep learning framework. As the latest iteration of the YOLO series, it maintains the real-time detection characteristics of the YOLO series and can achieve a high frame rate (FPS) even on lower hardware configurations. On this basis, many improvements have been made, including a more complex network architecture, a more optimized training process, and a more powerful feature extraction capability. Through a deeper and more complex network structure and improved training techniques, YOLOv8 has greatly improved the accuracy of detection while maintaining high speed. At the same time, YOLOv8 introduces an improved multi-scale prediction technology that can better detect objects of different sizes.

[0060] Step S3: logically reconstruct the detection layer operator network structure of the YOLO model.

[0061] The logical reconstruction of the detection layer operator network structure of the YOLO model mainly focuses on the CPU inference network to perform inference on the detection layer operator of the YOLO model and perform logical optimization on the process of bounding box prediction and category prediction.

[0062] The data flow diagram of this process is as follows Figure 2 First, before executing the border prediction and the category prediction, the NPU inference result is used as the first input data, and the first input data is split into the first sub-data and the second sub-data according to the first preset dimension; wherein: the dimension of the first sub-data is greater than the dimension of the second sub-data, and the first sub-data is used for the border prediction, and the second sub-data is used for the category prediction.

[0063] For example, for the first input data input1DDR, the size is float[70*8400], the first input data is divided into first sub-data and second sub-data according to the 64 dimensions of the first preset dimension input1DDR, wherein the first sub-data is the first 64 dimensions of the first input data input1DDR, and the first 64 dimensions of the first input data are extracted, that is, the first sub-data is: box[x][y]=input1DDR[x×8400+y], and its scale is float[64,8400]; the second sub-data is the last 6 dimensions of the first input data input1DDR, the scale is float[6,8400], and the second sub-data is cls[x][y].

[0064] The process of border prediction specifically includes: firstly, the first sub-data is dimensionally upgraded and transposed to obtain the second data, wherein the dimension upgrading process is set_x=i / 16; set_y=i%16, and the second data obtained after transposition is trans[set_y][set_x][j]=box[i][j], and the scale of the second data is [16,4,8400].

[0065] Then, a softmax operation is performed on the second data to obtain the third data, that is, each value of the second data trans matrix is ​​exp, and the data trans_exp[i][j][k]=e is obtained. trans[i][j][k] ; Then divide by the sum in 16 dimensions to get the data Finally, the ratio of trans_exp[i][j][k] to total[j][k] is used to get softmax_box[j][k][i]. That is, softmax_box is the third data, and its scale is [16,4,8400].

[0066] Finally, a convolution operation is performed on the third data to obtain the final border prediction result, that is, the third data softmax_box is convolved on the dimension of 16 and multiplied by weight to obtain the data dfl_box, whose scale is [4,8400]. dfl_box is the final border prediction result, expressed as:

[0067]

[0068] From the above process, it can be seen that only 16 values ​​in the first sub-data input1DDR are needed to get a value of the final output dfl_box. However, these 16 values ​​are separated by 8400 data points. When directly calculating, 16 values ​​must be obtained each time, which does not meet the principle of spatiotemporal locality in programming.

[0069] In view of this, in some embodiments, the above process is further optimized, such as Figure 3 , Figure 4 As shown in Figure 3 The data structure flow chart of the DFL operator before optimization is shown. Figure 4The optimized DFL operator data structure flow chart is shown. Specifically, in the above-mentioned border prediction process, before the softmax operation, the second data is pre-dimensionalized and the dimension-reduced data is transposed to obtain the fourth data, that is, the second data trans[16,4,8400] is dimensionally reduced to data down_trans[16,4*8400], that is, trans[i][j][k]=down_trans[i][j×8400+k]; then down_trans is further transposed to down_trans2, and down_trans2 is used as the fourth data, that is, trans[i][j][k]=

[0070] down_trans2[j×8400+k][i]. After obtaining the fourth data, the above softmax operation can be further repeated on the fourth data to obtain total[j][k], the third data softmax_box[j][k][i] and the bounding box prediction result dfl_box[j][k] again. Through the above conversion, the second data is first converted into the fourth data, and every 16 adjacent fourth data down_trans2 values ​​can obtain one dfl_box value, which greatly reduces the memory access amount.

[0071] For category prediction, the operation of category prediction is relatively simple compared to the operation of bounding box prediction, such as Figure 2 As shown, the second sub-data cls is input into a prediction model based on the sigmoid function, and the final category prediction result is output.

[0072] Step S4: perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

[0073] In this process, we first need to build a heterogeneous programming test framework to test the accuracy and inference speed improvement of the developed operators. Then, we perform BANGC operator programming and SIMD multi-core acceleration calculation on the reconstructed YOLO model detection layer in single-core and multi-core situations.

[0074] like Figure 5As shown in the figure, it shows the memory space structure diagram of the deep learning processor MLU270. The memory of the MLU deep learning processor is divided into two parts: on-chip memory and off-chip memory. Among them, the off-chip memory DRAM space is very large, and the address can be aligned to 32B, while the on-chip memory NRAM is only 512KB. The input data of the operator is 70*8400*4B≈2.25MB, which is much larger than 512KB, and it is impossible to copy all the data into the on-chip memory area for storage at one time. Therefore, after comprehensive consideration, the slices are used for calculation, and the size of each slice is set to LINESIZE=640*4B. At the same time, the on-chip memory data is reused to reduce the required memory space, but at this time, the temporal and spatial consistency of the data should be guaranteed as much as possible.

[0075] In the case of a single core, after programming the operator, it is found that the effect is general and the inference speed is slow. This is because after obtaining the fourth data in the above step S3, the above softmax operation is further repeated on the fourth data to obtain total[j][k], the third data softmax_box[j][k][i] and the border prediction result dfl_box[j][k], that is:

[0076] In the process of calculating the total data and the dfl_box data, two merge and add operations are performed. In the related art, a large loop is used to complete it, but the loop cannot fully exert the performance of the deep learning processor. In this regard, in this embodiment, the collect mask collector is used to rearrange the fourth data, and the addition operation is performed on the third data instead of the convolution operation. Specifically: Figure 6 As shown, for the generation of total data, two masks are introduced, the first mask mask0 and the second mask mask1. When the first mask is used, the value of the corresponding fourth data down_trans2 array is discarded, and when the second mask is used, the value of the corresponding fourth data down_trans2 array is retained. In the two new arrays generated in this way, each value of the first array is a number whose original array value address mod2 is 0, and each value of the second array is a number whose original array value address mod2 is 1. The two arrays are added to obtain the result of merging and summing every two numbers in the original array. Repeat this three times to obtain the result of merging and summing every 16 numbers in the original array. For dfl_box, it is necessary to multiply the original array by i first, and then use the collect mask collector to perform mask addition to obtain the final result. In this way, BANGC operator programming and SIMD multi-core accelerated computing are implemented for the reconstructed YOLO model detection layer in a single-core case.

[0077] For multi-core, it is similar to the single-core case, but the fourth data needs to be decomposed into multiple grouped data in advance and distributed to each core in the multi-core, so that each core can execute according to the operator acceleration process of the single core, that is, each core uses the collect mask collector to rearrange the grouped data corresponding to the core, and uses addition operation instead of convolution operation. In addition, see further Figure 5 As shown in , in the case of multiple cores, the inter-core shared memory SRAM is used instead of the global memory GDRAM.

[0078] In summary, the reasoning acceleration method of the embedded edge intelligent computing system disclosed by the present invention integrates CPU and NPU processors, builds the basic heterogeneous architecture of edge computing as the basis, and configures the model offline deployment framework, which supports the deployment of the trained YOLO model on the NPU processor. Then, the detection layer operator network structure of the YOLO model is logically reconstructed, and the operator programming and multi-core acceleration calculation are performed on the reconstructed YOLO model detection layer to improve the reasoning speed. This method realizes the construction of a high-computing power and strong real-time embedded intelligent computing system that meets the edge-side deployment application. Experimental verification shows that through this method, the operator speed is finally increased from 24.78ms to 5.36ms (single-core), 1.53ms (quad-core), and 0.383ms (16 cores). Under quad-core, the operator speed is increased by 16.2 times, and the theoretical reasoning speed of the entire system is increased from 33fps to 135fps, which has a good practical promotion prospect.

[0079] In addition, it should be noted that the inference acceleration method of the embedded edge intelligent computing system disclosed in the present invention is not limited to being executed in the order of the above steps S1-S4. It can reorder, add or delete steps. As long as the expected results of the technical solution of the present invention can be achieved, the present invention is not limited here.

[0080] In addition, corresponding to the embodiment of the inference acceleration method of the aforementioned embedded edge intelligent computing system, the present invention also provides an embodiment of the inference acceleration device of the embedded edge intelligent computing system.

[0081] See also Figure 7 , an embodiment of the present invention provides an inference acceleration device 200 for an embedded edge intelligent computing system, the device at least comprising:

[0082] The framework configuration module 210 is used to integrate the CPU and NPU processors to build a basic heterogeneous architecture for edge computing; and to configure a model offline deployment framework that supports deploying the trained YOLO model on the NPU processor.

[0083] The network optimization module 220 is used to logically reconstruct the detection layer operator network structure of the YOLO model.

[0084] The acceleration configuration module 230 is used to perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

[0085] Technical personnel in the relevant field can clearly understand that for the convenience and conciseness of describing the inference acceleration method of the embedded edge intelligent computing system, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0086] According to an embodiment of the present invention, the present invention also provides a machine-readable medium, which may be a tangible medium, which may contain or store a program for use by an instruction execution system, apparatus or device or for use in conjunction with an instruction execution system, apparatus or device, and the computer program, when executed by a processor, implements the steps of the above-mentioned inference acceleration method of the embedded edge intelligent computing system.

[0087] In addition, the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0088] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for accelerating reasoning in an embedded edge intelligent computing system, characterized in that: Include: Integrate CPU and NPU processors to build a basic heterogeneous architecture for edge computing; Configure the model offline deployment framework, which supports deploying the trained YOLO model on the NPU processor; Logically reconstructing the detection layer operator network structure of the YOLO model; Perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

2. The method according to claim 1, characterized in that The model offline deployment framework is configured with: NPU inference thread: used to send the preprocessed image data to the NPU inference network for inference, where the NPU inference network only performs inference to the layer before the detection layer of the YOLO model to obtain the NPU inference result.

3. The method according to claim 2, characterized in that The detection layer operator network structure of the YOLO model is logically reconstructed, including: Use the CPU inference network to infer the detection layer operators of the YOLO model and perform bounding box prediction and category prediction.

4. The method according to claim 3, characterized in that Before performing bounding box prediction and category prediction, it also includes: Using the NPU inference result as first input data, and splitting the first input data into first sub-data and second sub-data according to a first preset dimension; Wherein: the dimension of the first sub-data is greater than the dimension of the second sub-data, and the first sub-data is used for border prediction, and the second sub-data is used for category prediction.

5. The method according to claim 4, characterized in that The process of border prediction includes: Perform dimension upgrading and transposition on the first sub-data to obtain second data; Perform a softmax operation on the second data to obtain third data; A convolution operation is performed on the third data to obtain a final border prediction result.

6. The method according to claim 5, characterized in that Before the softmax operation, the second data is pre-dimensionalized, and the dimension-reduced data is transposed to obtain fourth data.

7. The method according to claim 4, characterized in that The process of category prediction includes: The second sub-data is input into a prediction model based on the sigmoid function, and a final category prediction result is output.

8. The method according to claim 5, characterized in that In single-core and multi-core cases, BANGC operator programming and SIMD multi-core accelerated computing are performed on the reconstructed YOLO model detection layer.

9. The method according to claim 8, characterized in that In the single-core case, the fourth data is rearranged using the collect mask collector. And an addition operation is performed on the third data instead of a convolution operation.

10. The method according to claim 8, characterized in that In the case of multiple cores, the fourth data is decomposed into multiple grouped data and distributed to each core in the multiple cores. Each core executes: using the collect mask collector to rearrange the grouped data corresponding to the core, and using addition operation instead of convolution operation.

11. The method according to claim 8, characterized in that In the case of multiple cores, the inter-core shared memory SRAM is used instead of the global memory GDRAM.

12. The method according to claim 1, characterized in that The YOLO model adopts YOLOv8.

13. An inference acceleration device for an embedded edge intelligent computing system, characterized in that: Include: Framework configuration module, used to integrate CPU and NPU processors to build a basic heterogeneous architecture for edge computing; And, configure the model offline deployment framework, which supports deploying the trained YOLO model on the NPU processor; A network optimization module, used to logically reconstruct the detection layer operator network structure of the YOLO model; The acceleration configuration module is used to perform operator programming and multi-core accelerated computing on the reconstructed YOLO model detection layer.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the inference acceleration method of the embedded edge intelligent computing system described in any one of claims 1 to 12 are implemented.

Images