Image processing method and system of deep learning network model based on edge calculation

Abstract

The invention discloses an image processing method and system of a deep learning network model based on edge calculation, and the method comprises the steps: obtaining an original deep learning network model, carrying out the vector conversion processing and row vector calculation processing of the original deep learning network model, and obtaining a first optimization result; performing quantitative acceleration processing on the first optimization result to obtain a second optimization result; performing indirect buffer area construction processing on the second optimization result to obtain a third optimization result; performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; and obtaining target image data, inputting the target image data into the target deep neural network model, and outputting a target image analysis result. According to the method, the deep learning network model deployed at the edge end can be accelerated, so that the processing efficiency and accuracy of the image data at the edge end are improved.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to an image processing method, system, terminal and computer-readable storage medium based on a deep learning network model of edge computing. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, edge computing devices are playing an increasingly important role in processing real-time, large-scale data (such as image recognition, natural language processing, and sensor data analysis). However, the limited computing resources and storage capacity of edge devices cannot meet the high performance requirements of deep learning models.

[0003] To solve the above problems, the prior art proposes to use CUDA (Compute Unified Device Architecture) to accelerate the deep learning network on the GPU (Graphic Processing Unit). However, the prior art method of accelerating the processing efficiency of the deep learning network model through CUDA at the edge has the problems of model optimization difficulties and low computing performance, which leads to low efficiency and low accuracy of the output data analysis results.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide an image processing method, system, terminal and computer-readable storage medium based on a deep learning network model of edge computing, aiming to solve the problem in the prior art that the processing efficiency of the deep learning network model accelerated by CUDA at the edge is difficult to optimize and the computing performance is low, resulting in low efficiency and low accuracy of the output data analysis results.

[0006] To achieve the above object, the present invention provides an image processing method based on a deep learning network model of edge computing, and the image processing method based on a deep learning network model of edge computing comprises the following steps:

[0007] Acquire an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result;

[0008] Performing quantization acceleration processing on the first optimization result to obtain a second optimization result;

[0009] Performing indirect buffer construction processing on the second optimization result to obtain a third optimization result;

[0010] Performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0011] Acquire target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0012] Optionally, the image processing method of the deep learning network model based on edge computing, wherein the obtaining of the original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model, and obtaining a first optimization result, specifically includes:

[0013] Acquire an original deep learning network model and target three-dimensional data, and input the target three-dimensional data into the original deep learning network model;

[0014] Obtaining an image vectorization function, and rewriting the image vectorization function using parallel computing device architecture programming to obtain a rewritten image vectorization function;

[0015] Performing vector conversion processing on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix;

[0016] A general matrix multiplication optimization strategy is used to perform row vector calculation processing on the target two-dimensional matrix to obtain a first optimization result.

[0017] Optionally, the image processing method based on the deep learning network model of edge computing, wherein the quantization acceleration processing is performed on the first optimization result to obtain the second optimization result, specifically includes:

[0018] Obtaining a quantized neural network package, and constructing a microkernel primitive based on the quantized neural network package;

[0019] The first optimization result is quantized and accelerated according to the microkernel primitives using high-performance matrix multiplication to obtain a second optimization result.

[0020] Optionally, the image processing method based on the deep learning network model of edge computing, wherein the indirect buffer construction processing is performed on the second optimization result to obtain a third optimization result, specifically including:

[0021] Obtaining an input buffer of the original deep learning network model, and constructing an indirect buffer according to the input buffer using an indirect convolution algorithm;

[0022] A pointer simulation method is used to perform memory access simulation processing on the second optimization result according to the indirect buffer to obtain a third optimization result.

[0023] Optionally, the image processing method of the deep learning network model based on edge computing, wherein the deep learning framework reconstruction processing is performed on the original deep learning network model according to the third optimization result to obtain a target deep neural network model, specifically includes:

[0024] Acquire deep learning reasoning parameters, and perform matrix vectorization processing on the third optimization result according to the deep learning reasoning parameters to obtain a vectorized result, wherein the deep learning reasoning parameters include a workspace size, an input image matrix format, and a quantized floating-point operation;

[0025] The vectorization result is subjected to deep learning framework reconstruction processing according to the parallel computing device architecture programming to obtain a target deep neural network framework.

[0026] Optionally, the image processing method based on the deep learning network model of edge computing, wherein the step of acquiring target image data, inputting the target image data into the target deep neural network model, and outputting the target image analysis result, specifically includes:

[0027] Acquire target image data, and input the target image data into the target deep neural network model;

[0028] The target image data is subjected to image analysis processing by the target deep neural network model, and a target image analysis result is output.

[0029] Optionally, in the image processing method based on the deep learning network model of edge computing, the general matrix multiplication optimization strategy includes address offset processing, simplified loop processing and memory alignment processing.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides an image processing system based on a deep learning network model of edge computing, wherein the image processing system based on a deep learning network model of edge computing includes:

[0031] A first optimization processing module is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result;

[0032] A second optimization processing module, used for performing quantization acceleration processing on the first optimization result to obtain a second optimization result;

[0033] A third optimization processing module, used for performing indirect buffer construction processing on the second optimization result to obtain a third optimization result;

[0034] A deep learning framework reconstruction module, used to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0035] The image result output module is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image processing program based on a deep learning network model based on edge computing stored in the memory and runnable on the processor, wherein the image processing program based on the deep learning network model based on edge computing, when executed by the processor, implements the steps of the image processing method based on the deep learning network model based on edge computing as described above.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program based on a deep learning network model based on edge computing, and when the image processing program based on the deep learning network model based on edge computing is executed by a processor, the steps of the image processing method based on the deep learning network model based on edge computing as described above are implemented.

[0038] In the present invention, the original deep learning network model is obtained, and the original deep learning network model is subjected to vector conversion processing and row vector calculation processing to obtain a first optimization result; the first optimization result is subjected to quantization acceleration processing to obtain a second optimization result; the second optimization result is subjected to indirect buffer construction processing to obtain a third optimization result; the original deep learning network model is subjected to deep learning framework reconstruction processing according to the third optimization result to obtain a target deep neural network model; the target image data is obtained, and the target image data is input into the target deep neural network model, and the target image analysis result is output. The present invention is based on technical processing such as GEMM (general matrix multiplication) vector calculation, CUDA (parallel computing device architecture) quantization programming and TensorRT (deep learning reasoning parameter) tensor reasoning, which effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization in the prior art of CUDA accelerated deep learning network model in edge computing. At the same time, the real-time calculation, fast and efficient learning reasoning tasks of deep learning models in edge computing devices are realized, which effectively improves the processing efficiency and accuracy of image data at the edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0040] Figure 2 It is a schematic diagram of a CUDA accelerated deep learning framework implementation process of a preferred embodiment of an image processing method of a deep learning network model based on edge computing of the present invention;

[0041] Figure 3 It is a schematic diagram of the communication and calculation methods of edge computing and cloud computing in a preferred embodiment of the image processing method based on the deep learning network model of edge computing of the present invention;

[0042] Figure 4 It is a schematic diagram of the convolution matrix calculation of im2col in a preferred embodiment of the image processing method based on the deep learning network model of edge computing of the present invention;

[0043] Figure 5 It is a schematic diagram of the comparison of the GEMM algorithm using optimized matrix multiplication in a preferred embodiment of the image processing method based on the deep learning network model of edge computing of the present invention;

[0044] Figure 6 It is a schematic diagram of the difference between GPU and CPU programming in a preferred embodiment of the image processing method based on the deep learning network model of edge computing of the present invention;

[0045] Figure 7 It is a schematic diagram of matrix multiplication to convolution of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0046] Figure 8 It is a schematic diagram of a deep convolution implementation method of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0047] Fig. 9 is a schematic diagram of an NVIDIA inference stack of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0048] Fig.10 It is a structural diagram of a preferred embodiment of an image processing system based on a deep learning network model of edge computing of the present invention;

[0049] Fig.11 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] With the rapid development of artificial intelligence and Internet of Things technologies, edge computing devices are playing an increasingly important role in processing real-time, large-scale data. However, edge devices usually have limited computing resources and storage capabilities, which makes it difficult to meet the high-performance requirements of deep learning models. To solve this problem, using CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model, generally referred to as unified computing device architecture / parallel computing device architecture) to accelerate deep learning networks on NVIDIA (NVIDIA Corporation, graphics chip and motherboard chip) GPU (graphics processing unit) has become an effective technical approach. CUDA is a parallel computing platform and programming model launched by NVIDIA that allows developers to use GPUs for efficient parallel computing general processing, including accelerated computing deep learning tasks.

[0052] The two core technologies for accelerating deep learning with CUDA programming on edge devices are CUDA programming and TensorRT reasoning. 1. Through CUDA programming, developers can directly access the hardware resources of the GPU and use the parallel computing capabilities of the GPU to accelerate the training and reasoning process of deep learning models. At present, software frameworks for building and training deep learning models, including mainstream deep learning frameworks such as TensorFlow, PyTorch, and Keras, all support CUDA acceleration. 2. NVIDIA's high-performance deep learning reasoning optimizer TensorRT (a software development kit for optimizing trained deep learning models for high-performance reasoning) can further accelerate the reasoning process of deep learning models. TensorRT supports importing models from multiple deep learning frameworks and provides a series of optimization technologies, such as layer fusion, dynamic memory management, etc., to improve reasoning speed and efficiency.

[0053] The shortcomings of existing technologies include the following aspects: 1. High learning cost: CUDA technology mainly relies on NVIDIA's GPU hardware. CUDA programming has high requirements for developers and requires knowledge of parallel computing, GPU architecture, etc. The learning cost is high, and beginners cannot directly use CUDA programming for acceleration. 2. Model optimization challenges: It is challenging to optimize deep learning models for deployment on edge devices. Techniques such as model quantization, pruning, and compression are used to reduce model size and computing requirements, but these techniques sometimes lead to reduced performance or accuracy. At the same time, problems such as version control, model updates, and monitoring need to be effectively addressed. 3. Edge device resource limitations: Although CUDA technology can achieve efficient parallel computing on GPUs, edge devices usually have limited computing resources and storage capacity, which cannot meet the requirements of real-time and large-scale data processing. It leads to certain limitations when edge devices run CUDA-based deep learning models, which may lead to reduced accuracy or performance. 4. Network latency and data security: In edge computing environments, devices usually need to communicate through the network, so it is crucial to ensure the security and privacy of this data during the inference process. At the same time, if a large amount of data needs to be transmitted between the cloud and edge devices, this may introduce network latency and communication overhead, affecting the real-time and accuracy of the deep learning model and leading to increased bandwidth and storage costs.

[0054] In summary, although the CUDA-accelerated deep learning network model framework based on edge computing devices has significant technical advantages, it also has some disadvantages and challenges. Future research and development needs to focus on how to reduce learning costs, improve model optimization performance, optimize edge device resources, and reduce network latency and data security, so as to promote the widespread application and development of deep learning technology based on edge computing devices.

[0055] To solve the above problems, the present invention provides a CUDA acceleration framework and process for deep learning on the edge side. The specific steps include: 1. GEMM (General Matrix Multiplication) is used to implement matrix multiplication. It has its own set of rules and strategies for matrix multiplication. Matrix multiplication is the multiplication of elements one by one) and im2col (image to column, sliding template by column, then converting the data contained in each window into column vectors, and finally arranging them into new matrices by column); 2. GEMM optimization strategy; 3. QNNPACK (Quantized Neural NetworkPACKage, Quantized Neural Network Package) quantization processing strategy; 4. Create indirect buffer to implement convolution operation; 5. TensorRT (high-performance deep learning inference optimizer) tensor conversion; 6. CUDA programming implementation. The present invention realizes the real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices. Based on technical processing such as GEMM vector calculation, CUDA quantization programming and TensorRT tensor reasoning, it effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization of existing edge computing CUDA accelerated deep learning network models.

[0056] The image processing method based on the deep learning network model of edge computing described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the image processing method based on the deep learning network model of edge computing includes the following steps:

[0057] Step S10: obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result.

[0058] GEMM matrix multiplication and im2col (image to column, image vectorization, sliding template by column, then converting the data contained in each window into column vectors, and finally arranging them into new matrices by column): Taking advantage of GEMM matrix calculation (the basic method of GEMM matrix is ​​1. Divide the output into several 4×4 sub-blocks to improve the reuse of input data, while using a large number of registers to reduce memory access; 2. Vectorize memory access and calculation; 3. Eliminate pointer calculation; 4. Reorganize memory to address continuity, etc.), and refer to the implementation method of im2col to convert the input three-dimensional data into a two-dimensional matrix (that is, convert the length, width and number of channels of the image into two dimensions), re-implement the main function logic of the im2col algorithm in CUDA GPU programming (using the strategy and ideas of GPU programming, rewrite the im2col algorithm including the logic and corresponding function methods of CPU matrix multiplication). Among them, im2col will expand the input data to fit the filter (weight). Specifically, for the input data, the area (3D square) where the filter is applied is horizontally expanded into 1 column. im2col will perform expansion processing in all places where the filter is applied.

[0059] GEMM's optimization strategy: Use row vector calculation method (calculate row by row, not column by column or matrix by element), replace and optimize matrix multiplication in registers, reduce access to data in non-continuous memory areas, calculate logical operations (addition, subtraction, multiplication, division, etc.) of 4*4 matrices at one time, and optimize the forward and backward feedback networks and total inference time of the neural network model.

[0060] Specifically, an original deep learning network model and target three-dimensional data are obtained, and the target three-dimensional data are input into the original deep learning network model; an image vectorization function is obtained, and the image vectorization function is rewritten using parallel computing device architecture programming to obtain a rewritten image vectorization function; vector conversion is performed on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix; a general matrix multiplication optimization strategy is used to perform row vector calculation on the target two-dimensional matrix to obtain a first optimization result, wherein the general matrix multiplication optimization strategy includes address offset processing, simplified loop processing, and memory alignment processing.

[0061] like Figure 3 As shown, edge devices perform computations locally, closer to the data source or the “edge” of the network rather than relying solely on centralized cloud servers. Figure 3 This includes smartphones, IoT devices, and edge servers. Figure 3 As shown in the example, Figure 3A in the figure connects the drone (sensing terminal), edge computing device (computing terminal) and 4G / 5G communication base station (communication terminal). Figure 3 The B in the diagram connects drones, cloud servers, and communication base stations. By leveraging CUDA acceleration on edge devices, deep learning inference tasks can be performed faster and more efficiently, enabling real-time or near-real-time data processing without relying on remote servers. This integration allows tasks such as image recognition, natural language processing, and sensor data analysis to be performed locally, which is critical for applications with low latency or limited connectivity.

[0062] GEMM matrix multiplication and im2col vector conversion: In the original CUDNN (CUDA Deep NeuralNetwork, GPU acceleration library for deep neural networks) framework, NVIDIA integrates deep learning matrix calculations without providing layer abstraction, but provides lower-level computer primitive semantics to simplify integration with existing deep learning frameworks. These include classic Fourier transform (FFT), Volker Strassen, and Winograd. 1. Fourier transform and fast Fourier transform are commonly used calculation methods in classic image processing, but they are usually not used in ConvNet, mainly because the convolution templates in ConvNet are usually small, such as 3×3, etc. In this case, the time overhead of FFT is even greater, so FFT is rarely used to implement convolution in CNN; 2. Volker Strassen is only more advantageous when the convolution kernel is large. At present, this method has not been seen in open source frameworks, which is also limited by the size of the convolution kernel; 3. In most scenarios, the Winograd method shows a greater advantage, and this method is currently used to calculate convolution in CUDNN.

[0063] The neural networks used for computer vision in the prior art spend most of their reasoning time on convolution and fully connected operators. These operators are closely related to matrix multiplication: fully connected operators and 1×1 convolutions are directly mapped to matrix multiplications, and convolutions with larger kernels can be decomposed into a combination of a memory layout transformation called im2col and matrix multiplication. Therefore, the problem of effective reasoning in convolutional neural networks is largely a problem of effective implementation of matrix multiplication, which is also called GEMM in linear algebra libraries. With the adoption of the im2col+GEMM synthesis method in the caffe (Convolutional Architecture for Fast Feature Embedding, convolutional neural network framework) framework, almost all mainstream computing frameworks including Caffe, MXNet and TensorFlow have implemented this method. This method converts the entire convolution process into a GEMM process, and GEMM is extremely optimized in various BLAS libraries, and generally speaking, it is faster.

[0064] like Figure 4 As shown in the figure, im2col converts the input three-dimensional data into a two-dimensional matrix, which allows the convolution calculation to be expressed as the multiplication of two two-dimensional matrices, thereby making full use of the optimized GEMM library to accelerate the convolution calculation for each platform. The cost of im2col using GEMM to calculate convolution is additional memory overhead, and the input will use an additional x times of memory.

[0065] Among them, CUDA programming is used to rewrite im2col, as shown in the following function body:

[0066]

[0067]

[0068] GEMM optimization strategy: In order to solve the above problems, the basic method is to divide the output (the output is a two-dimensional matrix, which refers to the output of Im2col, that is, the result of multiplying the input image and the convolution kernel. The storage format and representation are the same as the original im2col, but the logic code is rewritten with CUDA) into several 4×4 sub-blocks. When these elements are single-precision floating-point numbers, the memory size is 16 bytes, so the reuse of input data can be improved. At the same time, registers are used extensively to reduce memory access. The purpose is to directly load 4 elements instead of using 4 discontinuous memory accesses 4 times; vectorize memory access and calculation. In the process of CUDA vector calculation, row vector calculation is used to implement 4 vector accesses to load into registers to realize the calculation of the original 16 elements; eliminate pointer calculation, and use reference calculation for the pointer of input matrix B; reorganize memory to address continuously, etc.

[0069] Now we calculate a 4x 4 block of C at a time (a row of A multiplied by a column of B to get an element of C, where A is the input, B is the weight, and C is the output) in order to use vector instructions and vector registers efficiently. The idea is as follows: There are special instructions in the SSE3 instruction set that allow two "multiplication-accumulation" operations (two multiplications and two additions) to be performed per clock cycle, for a total of four floating-point operations per clock cycle. To use these, the data must be placed in a "vector register". There are sixteen of them, each of which can hold two double-precision numbers. Therefore, 32 double-precision numbers can be kept in a register. Sixteen of them will be used to hold the elements of C, that is, 4x 4 blocks. Among them, the horizontal axis is the matrix size. For easier operation, the MNK value is assumed to be the same, and the vertical axis is gflops. The original version of the matrix multiplication is less than 1 gflops, and it will decay as the matrix size increases, such as Figure 5 As shown (representing the number of operations Gflops before and after GEMM optimization respectively).

[0070] Its core optimization strategies include the following three aspects:

[0071] 1. Avoid multiplication:

[0072] for(p=0;p <k;p++){

[0073] a_0p_reg = A(0, p);

[0074] c_00_reg+=a_0p_reg*B(p, 0);

[0075] c_01_reg+=a_0p_reg*B(p, 1);

[0076] c_02_reg+=a_0p_reg*B(p, 2);

[0077] c_03_reg+=a_0p_reg*B(p, 3);

[0078] }

[0079] Each calculation of B(p, 0) requires multiplication (B(p, 0) represents the element position of matrix B, p rows, 0 columns). We can use a pointer to save the address of B(p, 0) and offset it when we need B(p, 1). For example, the original text is in column-major order, and the modified code snippet is as follows:

[0080] for(p=0;p <k;p++){

[0081] a_0p_reg=A(0,p);

[0082] c_00_reg+=a_0p_reg**bp0_pntr++;

[0083] c_01_reg+=a_0p_reg**bp1_pntr++;

[0084] c_02_reg+=a_0p_reg**bp2_pntr++;

[0085] c_03_reg+=a_0p_reg**bp3_pntr++;

[0086] }

[0087] Therefore, 4 multiplications are saved (the original code needs to multiply and add each time, but now it uses pointer reference and can be directly referenced according to the storage address. It needs to take the value each time, a total of 4*4 area, saving 4 multiplication operations).

[0088] 2. Simplify the loop: Simple loop structure is more conducive to compiler optimization.

[0089] 3. Memory alignment: Assume that the cache line is 32B. If the size of the data to be accessed is 64B and the address is 0x80000001, then three cache mapping table entries are required; if the address is 0x80000000, only two entries are required. Memory alignment improves the cache hit rate in disguise.

[0090] Taking the classification deep learning network AlexNet (input size is 128x3x224x224) as an example, the comparison of its optimized front and back feedback networks and total inference time is shown in Table 1:

[0091] Table 1: Comparison of inference speed of each framework of AlexNet network

[0092] Library Time(ms) forward(ms) backward(ms) CuDNN-fp16 71 25 46 CuDNN-fp32 81 27 53 CuDNN+Caffe 231 70 161 Caffe 324 121 203 Torch-7 342 132 210

[0093] By taking advantage of the respective advantages of GEMM matrix multiplication and im2col vector calculation, and reconstructing the convolution operation and accumulation calculation using the logic of row vector GPU programming, it is possible to achieve fast calculation of the underlying operation logic (addition, subtraction, multiplication, division, etc.) and solve the memory usage problem.

[0094] Step S20: performing quantization acceleration processing on the first optimization result to obtain a second optimization result.

[0095] QNNPACK quantization processing strategy: Use PDOT microkernel primitives (panel dot product, which is a matrix multiplication method, and the primitive is a unit) to implement high-performance convolution matrix multiplication, and use quantized GPU (CUDA) programming to implement memory copy logic, realize multiplication and accumulation, and reduce the limitations of memory and cache bandwidth on convolution layer operations (CUDA computing is GPU computing, and the GPU has its own computing logic and implementation framework. The logic code in the CPU can be rewritten according to this framework).

[0096] It is possible to implement the entire matrix multiplication on the dot product primitive, but such an implementation is too inefficient. In a dot product, each multiply-add operation requires uploading two elements, and on current processors, this implementation is limited by memory and cache bandwidth rather than the computational power of the multiply-add unit. But a small modification - that is, calculating the dot product of several rows of A and several rows of B at the same time - greatly improves performance.

[0097] The modified primitive loads the MR of A and the NR of B, and performs the MRxNR multiplication-accumulation operation. The maximum values ​​of MR and NR are limited by the number of integers and other details of the processor architecture. However, in most modern systems, these maximum values ​​are large enough to make the operation limited only by calculation. All high-performance matrix multiplication implementations are built on this primitive, which is usually called the PDOT (panel dot product) microkernel (that is, the original multiplication required one calculation, which would be limited by memory and cache, because each calculation required space to store one element. Now that the calculation is implemented in the primitive, it can be calculated directly and the calculation result can be output to memory and cache without the need for intermediate storage process).

[0098] Specifically, a quantized neural network package is obtained, and a microkernel primitive is constructed according to the quantized neural network package; high-performance matrix multiplication is used to perform quantized acceleration processing on the first optimization result according to the microkernel primitive to obtain a second optimization result.

[0099] QNNPACK quantization processing strategy: Caffe2Go uses a kernel library called NNPACK, which implements a progressively fast convolution algorithm based on Winograd transform or fast Fourier transform, allowing convolution calculations to use several times fewer multiplication operations than direct algorithms. For example, a 3×3 convolution can only be 2 times slower than a 1×1 convolution, while the direct algorithm is 9 times slower.

[0100] QNNPACK (Quantized Neural Network PACKage) is part of Facebook's application and has been deployed on billions of mobile devices around the world. This new library (QNNPACK) can perform advanced computer vision tasks, such as running Mask R-CNN (an AI object detection algorithm) and DensePose (an AI pose estimation algorithm) in real time on mobile phones or implementing image classification in less than 100ms on performance-constrained mobile devices.

[0101] For example, A is the input, B is the weight, and C is the output. During inference, B never changes, so it does not take time to migrate to any convenient storage configuration. The MxK matrix A is multiplied by the KxN matrix B to obtain the MxN matrix C, where each element can be considered as the dot product of the row of A and the corresponding column of B.

[0102] It is possible to implement the entire matrix multiplication on the dot product primitive, but such an implementation is too inefficient. In a dot product, each multiply-add operation requires uploading two elements, and on current processors, this implementation is limited by memory and cache bandwidth rather than the computational power of the multiply-add unit. However, a small modification to calculate the dot products of several rows of A and several rows of B at the same time can greatly improve performance.

[0103] like Figure 6 As shown in the figure, by comparing CPU programming and GPU (CUDA) programming, we can see the difference in the implementation of memory copy logic.

[0104] The modified primitive loads the MR (a block of computational units or storage units with M rows and R columns, which is simply understood as the MR matrix) of A and the NR (N rows and R columns) elements of B, and performs the MRxNR multiply-accumulate operation. The maximum values ​​of MR and NR are limited by the number of integers and other details of the processor architecture. However, in most modern systems, these maximum values ​​are large enough to make the operation only limited by the calculation. All high-performance matrix multiplication implementations are built on this primitive, which is often called the PDOT (panel dotproduct) microkernel.

[0105] QNNPACK uses a linear quantization scheme compatible with the Android Neural Network API, which assumes that the quantized q[i] is represented as an 8-bit unsigned integer, and they are related to the real-valued representation r[i], as follows: r[i] = scale*(q[i]-zero_point); where scale is a positive floating point number and zero_point is an unsigned 8-bit integer, the same as q[i]. Using the QNNPACK quantization strategy further reduces the computational complexity of the network model and the resource consumption of floating-point calculations, making it possible to apply deep learning network models in edge computing devices while ensuring network accuracy.

[0106] Step S30: perform indirect buffer construction processing on the second optimization result to obtain a third optimization result.

[0107] Create an indirect buffer to implement convolution operations: Take advantage of the QNNPACK indirect buffer to avoid the problem of increased memory overhead introduced by im2col vectorization. At the same time, in the spatial zero padding position during spatial convolution, use an indirect pointer to point to the fixed address to solve the problem of complex memory address pointing calculation.

[0108] Specifically, the input buffer of the original deep learning network model is obtained, and an indirect buffer is constructed according to the input buffer using an indirect convolution algorithm; a pointer simulation method is used to perform memory access simulation processing on the second optimization result according to the indirect buffer to obtain a third optimization result.

[0109] like Figure 7 As shown, Figure 7 Detailed process of the indirect convolution algorithm workflow. Figure 7 The a in indicates that multiple inputs use the same input buffer. The indirect convolution algorithm builds the following Figure 7 b in Figure 7 The b in is the [indirect buffer] (Indirect Buffer), and the indirect buffer is the core of the indirect convolution algorithm. Figure 7 b in the figure, when the network is running, each time an output of size M×N (O rows and H columns) is calculated, where M is the vectorized size after treating OH×OW as one dimension. Generally, M×N is 4×4, 8×8 or 4×8. When calculating an output of size M×N, the corresponding input is taken out through the indirect buffer, and the weight is taken out to calculate the result. The calculation process is equivalent to calculating the matrix multiplication of M×K and K×N.

[0110] In the specific implementation process, the software execution process is divided into two parts:

[0111] 1. Preparation phase: load the model, configure the input buffer; rearrange the weights so that its memory layout is suitable for subsequent calculations; 2. Running phase: for each input, run [OH*OW / M][OC / N] core loops, each time using the GEMM method to calculate an M×N output.

[0112] The indirect buffer makes it possible to simulate memory access to the input through pointers. When the micro kernel that actually calculates the M×N output is actually running, there will be M pointers scanning the input (image or matrix, output of the previous layer).

[0113] The Im2col optimization algorithm first copies the input into a matrix, such as Figure 8 The indirect buffer used by the indirect convolution optimization algorithm actually stores pointers to the input (this is why the indirect convolution optimization algorithm requires the input memory address to be fixed). At runtime, these pointers can be used to simulate a matrix calculation process similar to Im2col.

[0114] The indirect convolution optimization algorithm solves three problems in convolution calculation. The first is the spatial vectorization problem, the second is the complex address calculation problem, and the third is the memory copy problem. Generally, when calculating convolution, it is necessary to fill the input with zeros (for the case where KH×KW is not 1×1). In this process, traditional methods will cause memory copies. The indirect buffer in the indirect convolution optimization algorithm can cleverly solve this problem through indirect pointers. When constructing the indirect buffer, an additional 1×IC (I-row C-column matrix) memory buffer is created and filled with zero values. For the positions in the space that need to be filled with zeros, the corresponding indirect pointer points to the buffer, so that the subsequent calculations are equivalent to having been filled with zeros.

[0115] In QNNPACK, the present invention implements a more efficient algorithm that does not transform the convolution input to fit the implementation of matrix multiplication, but instead adjusts the implementation of the PDOT microkernel to virtually perform the im2col transformation. Instead of copying the actual data from the input tensor to the im2col buffer, an indirect buffer is set up with pointers to the rows of input pixels involved in computing each output pixel. The matrix multiplication microkernel is also modified (Load pointer (matrix) to be computed from indirect buffer) to load pointers to the rows of the virtual matrix A from the indirect buffer, which is typically much smaller than the im2col buffer. In addition, if the memory location of the input tensor does not change between inference runs, the indirect buffer can also be initialized once with pointers to the input rows and then reused in multiple inference runs. It can be observed that the microkernel with an indirect buffer not only eliminates the overhead of the im2col transformation, but also performs slightly better than the matrix multiplication microkernel (probably because input rows are reused when computing different output pixels).

[0116] like Figure 8 As shown in the quantized MobileNetV2 benchmark, QNNPACK outperforms the most advanced implementations on various mobile phones by about 2 times. The main change is the change in convolution. For a 3×3 depth convolution implementation, each output row will be updated 9 times. QNNPACK calculates the results of all 3×3 convolution kernel rows and 3×3 input rows, accumulates them to the output row at one time, and then processes the next output row.

[0117] Step S40: Perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model.

[0118] TensorRT tensor conversion: Based on NVIDIA's own deep learning framework, it combines the important features of the TensorRT reasoner on the edge side to implement matrix vectorization operations and complete the reasoning reconstruction of each layer of the model according to factors such as its workspace size, input image matrix format, and quantized floating-point operations.

[0119] CUDA programming implementation: Based on the three characteristics of deep learning of the cuDNN acceleration library, and according to the foundations of GEMM vectorized calculation, QNNPACK quantization acceleration and TensorRT tensor conversion, the deep learning framework of CUDA programming (cuDNN acceleration library) is reconstructed.

[0120] Specifically, deep learning inference parameters are obtained, and matrix vectorization is performed on the third optimization result according to the deep learning inference parameters to obtain a vectorized result, wherein the deep learning inference parameters include a workspace size, an input image matrix format, and a quantized floating-point operation; deep learning framework reconstruction is performed on the vectorized result according to the parallel computing device architecture programming to obtain a target deep neural network framework.

[0121] TensorRT tensor conversion: TensorRT can allocate workspace memory for intermediate buffers inside the network. When it comes to execution, TensorRT can generally provide faster execution because it guarantees that the best execution path is chosen for the entire graph rather than just a sub-part. This obviously comes at the cost of such high engine creation time, so there are some deployment issues to consider, such as Fig. 9 As shown, Fig. 9 Three important parameters for deploying TensorRT for CUDA: 1. TensorRT workspace size; 2. NHWC format (N (sample) number of dimensions H height W width C number of channels); 3. FP16 and FP8 (16-bit floating point and 8-bit floating point).

[0122] The workspace size of TensorRT is an important parameter of TensorRT. Since TensorRT can rearrange the operations in the graph for optimization, more memory may be needed to store intermediate results. This value is controlled by the workspace size, telling the engine builder how many bytes TensorRT can allocate in addition to the engine weight and input. It makes sense to experiment with this parameter and trade off speed with the inference size.

[0123] Inputs in NHWC format are well suited for Tensor Core GPUs on NVIDIA. Since ONNX only supports the NCHW format, a trick must be used to enable NHWC as an input tensor. Set the input dimension to NHWC and insert a Transpose operation after the input removed by CUDA or TensorRT EP.

[0124] Operation precisions such as FP16 and FP8 are particularly important for optimal performance on the GPU. In TensorRT EP, they must be explicitly enabled during session creation using the following property: OrtTensorRTProviderOptions trt_options{};

[0125] trt_options.trt_fp16_enable=1;

[0126] trt_options.trt_int8_enable=1;

[0127] CUDA programming implementation: Currently, NVIDIA provides the cuDNN acceleration library to accelerate the operation of deep learning networks, which mainly includes: 1. Accelerated learning; 2. Expression operation graph API; 3. Fusion support.

[0128] 1. Accelerated learning: cuDNN provides kernels that target Tensor cores to provide the best available performance on compute-bound operations, and it provides heuristics for selecting the right kernel for a given problem size.

[0129] 2. Expression Graph API: Users define computations as graphs on tensors. The cuDNN library has both a direct C API and an open source C++ front end for ease of use, and most users choose the front end as the entry point to cuDNN.

[0130] 3. Fusion support: cuDNN supports the fusion of compute-bound and memory-bound operations. Common general-purpose fusion modes are usually implemented through runtime kernel generation. Specialized fusion modes are optimized using pre-written kernels.

[0131] CuDNN benchmarking was done using Torch bindings, the same can be done with Caffe bindings or bindings for any other library. This post is to clarify that Caffe (native) and Torch (native) are convolution kernels that come as default fallbacks. Some frameworks like TensorFlow and Chainer are benchmarked against CuDNN but it is not explicitly mentioned, so one might assume that these frameworks as a whole are faster than Caffe, which might not be the case.

[0132] The present invention realizes the reconstruction of the deep learning framework of CUDA programming (cuDNN acceleration library) based on GEMM vectorized calculation, QNNPACK quantization acceleration and TensorRT tensor conversion, including the rewriting of Conv, flatten, max_pool and other layer networks (Cuda implementation).

[0133] Take forward as an example

[0134]

[0135] Table 2: Edge intelligence device environment configuration

[0136]

[0137] Based on the features of the CUDNN acceleration library and the advantages of the TensorRT reasoner for tensor computing, the present invention uses CUDA programming to rebuild the implementation logic of each layer of the network model, which can more quickly realize the rapid transplantation and lightweight of different deep learning frameworks in edge devices.

[0138] Step S50: Acquire target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0139] Specifically, target image data is acquired and input into the target deep neural network model; image analysis processing is performed on the target image data through the target deep neural network model, and a target image analysis result is output.

[0140] After accelerating the neural network model deployed on the edge, a target deep neural network model will be obtained, and the accelerated target deep neural network model can parse and process the input target image data without being limited by the computing resources and storage capacity of the edge device. At the same time, the accelerated target deep neural network model can also improve the image processing efficiency and accuracy at the edge.

[0141] In addition, the input of the target deep neural network model in the present invention is not limited to image recognition, but can also be natural language processing data or sensor data, all of which can be efficiently processed at the edge through the target deep neural network model.

[0142] In summary, the present invention is based on the CUDA accelerated deep learning network model framework of edge computing devices, and relates to an acceleration method for the edge-side deep learning network framework in the field of smart cities and low-altitude economy. First, based on the framework acceleration principle of GEMM matrix multiplication and the efficient computing characteristics of im2col vectorized calculation, row calculation is used to replace matrix calculation to reduce the problem of memory occupancy. Secondly, the QNNPACK quantization processing strategy is adopted to optimize the problem of discontinuous memory pointing in row calculation, and the strategy of indirect cache specific pointer pointing is adopted to realize the replacement of deep convolution operation pairs. Finally, using the TensorRT reasoner and cuDNN acceleration library, CUDA programming is used to reconstruct the logic of data processing at each layer of the neural network, and the application of deep learning models of different frameworks in edge devices is realized. The present invention realizes the real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices, and effectively solves the problems of high learning cost, low computing performance of edge devices and difficult model optimization of existing edge computing CUDA accelerated deep learning network models.

[0143] Furthermore, if Fig.10As shown, based on the above-mentioned image processing method of the deep learning network model based on edge computing, the present invention also provides an image processing system based on the deep learning network model based on edge computing, wherein the image processing system based on the deep learning network model based on edge computing includes:

[0144] A first optimization processing module 51 is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result;

[0145] A second optimization processing module 52 is used to perform quantization acceleration processing on the first optimization result to obtain a second optimization result;

[0146] A third optimization processing module 53, configured to perform indirect buffer construction processing on the second optimization result to obtain a third optimization result;

[0147] A deep learning framework reconstruction module 54 is used to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0148] The image result output module 55 is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0149] Furthermore, if Fig.11 As shown, based on the above-mentioned image processing method and system based on the deep learning network model based on edge computing, the present invention also provides a terminal accordingly, and the terminal includes a processor 10, a memory 20 and a display 30. Fig.11 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0150] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, an image processing program 40 based on a deep learning network model of edge computing is stored on the memory 20, and the image processing program 40 based on a deep learning network model of edge computing can be executed by the processor 10, thereby realizing the image processing method based on a deep learning network model of edge computing in the present application.

[0151] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image processing method based on the edge computing deep learning network model.

[0152] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0153] In one embodiment, when the processor 10 executes the image processing program 40 based on the deep learning network model based on edge computing in the memory 20, the steps of the image processing method based on the deep learning network model based on edge computing as described above are implemented.

[0154] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program based on a deep learning network model based on edge computing, and when the image processing program based on the deep learning network model based on edge computing is executed by a processor, the steps of the image processing method based on the deep learning network model based on edge computing as described above are implemented.

[0155] In summary, the present invention provides an image processing method and related equipment based on a deep learning network model of edge computing, the method comprising: obtaining an original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model, and obtaining a first optimization result; performing quantization acceleration processing on the first optimization result to obtain a second optimization result; performing indirect buffer construction processing on the second optimization result to obtain a third optimization result; performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; obtaining target image data, and inputting the target image data into the target deep neural network model, and outputting the target image analysis result. The present invention is based on technical processing such as GEMM (general matrix multiplication) vector calculation, CUDA (parallel computing device architecture) quantization programming and TensorRT (deep learning reasoning parameter) tensor reasoning, which effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization in the prior art of CUDA accelerated deep learning network models in edge computing. At the same time, it realizes the real-time calculation, fast and efficient learning reasoning tasks of deep learning models in edge computing devices, effectively improving the processing efficiency and accuracy of image data at the edge.

[0156] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or terminal including the element.

[0157] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0158] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image processing method based on a deep learning network model of edge computing, characterized in that: The image processing method based on the deep learning network model of edge computing includes: Acquire an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result; Performing quantization acceleration processing on the first optimization result to obtain a second optimization result; Performing indirect buffer construction processing on the second optimization result to obtain a third optimization result; Performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; Acquire target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

2. The image processing method based on the deep learning network model of edge computing according to claim 1 is characterized in that: The obtaining of the original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model, and obtaining a first optimization result specifically includes: Acquire an original deep learning network model and target three-dimensional data, and input the target three-dimensional data into the original deep learning network model; Obtaining an image vectorization function, and rewriting the image vectorization function using parallel computing device architecture programming to obtain a rewritten image vectorization function; Performing vector conversion processing on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix; A general matrix multiplication optimization strategy is used to perform row vector calculation processing on the target two-dimensional matrix to obtain a first optimization result.

3. The image processing method based on the deep learning network model of edge computing according to claim 1, characterized in that: The performing quantization acceleration processing on the first optimization result to obtain the second optimization result specifically includes: Obtaining a quantized neural network package, and constructing a microkernel primitive based on the quantized neural network package; The first optimization result is quantized and accelerated according to the microkernel primitives using high-performance matrix multiplication to obtain a second optimization result.

4. The image processing method based on the deep learning network model of edge computing according to claim 1, characterized in that: The performing indirect buffer construction processing on the second optimization result to obtain a third optimization result specifically includes: Obtaining an input buffer of the original deep learning network model, and constructing an indirect buffer according to the input buffer using an indirect convolution algorithm; A pointer simulation method is used to perform memory access simulation processing on the second optimization result according to the indirect buffer to obtain a third optimization result.

5. The image processing method based on the deep learning network model of edge computing according to claim 2, characterized in that: The performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model specifically includes: Acquire deep learning reasoning parameters, and perform matrix vectorization processing on the third optimization result according to the deep learning reasoning parameters to obtain a vectorized result, wherein the deep learning reasoning parameters include a workspace size, an input image matrix format, and a quantized floating-point operation; The vectorization result is subjected to deep learning framework reconstruction processing according to the parallel computing device architecture programming to obtain a target deep neural network framework.

6. The image processing method based on the deep learning network model of edge computing according to claim 1, characterized in that: The acquiring target image data, inputting the target image data into the target deep neural network model, and outputting the target image analysis result specifically includes: Acquire target image data, and input the target image data into the target deep neural network model; The target image data is subjected to image analysis processing by the target deep neural network model, and a target image analysis result is output.

7. The image processing method based on the deep learning network model of edge computing according to claim 2, characterized in that: The general matrix multiplication optimization strategy includes address offset processing, loop simplification processing and memory alignment processing.

8. An image processing system based on a deep learning network model of edge computing, characterized in that: The image processing system based on the deep learning network model of edge computing includes: A first optimization processing module is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result; A second optimization processing module, used for performing quantization acceleration processing on the first optimization result to obtain a second optimization result; A third optimization processing module, used for performing indirect buffer construction processing on the second optimization result to obtain a third optimization result; A deep learning framework reconstruction module, used to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; The image result output module is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and an image processing program based on a deep learning network model based on edge computing, which is stored in the memory and can be run on the processor. When the image processing program based on the deep learning network model based on edge computing is executed by the processor, the steps of the image processing method based on the deep learning network model based on edge computing are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an image processing program of a deep learning network model based on edge computing. When the image processing program of the deep learning network model based on edge computing is executed by a processor, the steps of the image processing method of the deep learning network model based on edge computing are implemented as described in any one of claims 1 to 7.

Images