Image processing method and system based on deep learning network model of edge computing

Abstract

The present invention discloses an image processing method and system based on a deep learning network model of edge computing, the method comprising: obtaining an original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model to obtain a first optimization result; performing quantization acceleration processing on the first optimization result to obtain a second optimization result; performing indirect buffer construction processing on the second optimization result to obtain a third optimization result; performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; obtaining target image data, and inputting the target image data into the target deep neural network model, and outputting a target image analysis result. The present invention can accelerate the deep learning network model deployed at the edge to improve the processing efficiency and accuracy of image data at the edge.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and in particular to an image processing method, system, terminal and computer-readable storage medium based on a deep learning network model of edge computing. Background Art

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT), edge computing devices are playing an increasingly important role in processing real-time, large-scale data, such as image recognition, natural language processing, and sensor data analysis. However, the limited computing resources and storage capabilities of edge devices make it difficult to meet the high-performance requirements of deep learning models.

[0003] To solve the above problems, the existing technology proposes to use CUDA (Compute Unified Device Architecture) to accelerate deep learning networks on GPUs (Graphic Processing Units). However, the existing technology uses CUDA to accelerate the processing efficiency of deep learning network models at the edge, which has problems such as model optimization difficulties and low computing performance, resulting in low efficiency and low accuracy of the output data analysis results.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide an image processing method, system, terminal and computer-readable storage medium based on a deep learning network model of edge computing, aiming to solve the problem in the prior art of using CUDA to accelerate the processing efficiency of deep learning network models at the edge, which has optimization difficulties and low computing performance, resulting in low efficiency and low accuracy of the output data analysis results.

[0006] To achieve the above objectives, the present invention provides an image processing method based on a deep learning network model of edge computing, and the image processing method based on a deep learning network model of edge computing comprises the following steps:

[0007] Obtaining an original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model, and obtaining a first optimization result;

[0008] Performing quantization and acceleration processing on the first optimization result to obtain a second optimization result;

[0009] Performing indirect buffer construction processing on the second optimization result to obtain a third optimization result;

[0010] Performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0011] Obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0012] Optionally, the image processing method of the deep learning network model based on edge computing, wherein the obtaining of the original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model, and obtaining the first optimization result, specifically includes:

[0013] Acquire an original deep learning network model and target three-dimensional data, and input the target three-dimensional data into the original deep learning network model;

[0014] Obtaining an image vectorization function, and rewriting the image vectorization function using parallel computing device architecture programming to obtain a rewritten image vectorization function;

[0015] Performing vector conversion processing on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix;

[0016] A general matrix multiplication optimization strategy is used to perform row vector calculation processing on the target two-dimensional matrix to obtain a first optimization result.

[0017] Optionally, in the image processing method based on the deep learning network model of edge computing, the step of performing quantization acceleration processing on the first optimization result to obtain the second optimization result specifically includes:

[0018] Obtaining a quantized neural network package, and constructing a microkernel primitive based on the quantized neural network package;

[0019] High-performance matrix multiplication is used to perform quantized acceleration processing on the first optimization result according to the microkernel primitive to obtain a second optimization result.

[0020] Optionally, the image processing method based on the deep learning network model of edge computing, wherein the indirect buffer construction processing is performed on the second optimization result to obtain the third optimization result, specifically includes:

[0021] Obtaining an input buffer of the original deep learning network model, and constructing an indirect buffer according to the input buffer using an indirect convolution algorithm;

[0022] A pointer simulation method is used to perform memory access simulation processing on the second optimization result according to the indirect buffer to obtain a third optimization result.

[0023] Optionally, the image processing method of the deep learning network model based on edge computing, wherein the step of performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model, specifically includes:

[0024] Obtaining deep learning inference parameters, and performing matrix vectorization processing on the third optimization result according to the deep learning inference parameters to obtain a vectorized result, wherein the deep learning inference parameters include a workspace size, an input image matrix format, and a quantized floating-point operation;

[0025] The vectorization results are reconstructed into a deep learning framework according to the parallel computing device architecture programming to obtain a target deep neural network framework.

[0026] Optionally, the image processing method based on the deep learning network model of edge computing, wherein the step of acquiring target image data, inputting the target image data into the target deep neural network model, and outputting the target image analysis result, specifically includes:

[0027] Acquire target image data, and input the target image data into the target deep neural network model;

[0028] The target image data is subjected to image analysis processing by the target deep neural network model, and a target image analysis result is output.

[0029] Optionally, the image processing method of the deep learning network model based on edge computing, wherein the general matrix multiplication optimization strategy includes address offset processing, simplified loop processing and memory alignment processing.

[0030] In addition, to achieve the above-mentioned purpose, the present invention also provides an image processing system based on a deep learning network model of edge computing, wherein the image processing system based on a deep learning network model of edge computing includes:

[0031] A first optimization processing module is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result;

[0032] A second optimization processing module is used to perform quantization acceleration processing on the first optimization result to obtain a second optimization result;

[0033] A third optimization processing module, configured to perform indirect buffer construction processing on the second optimization result to obtain a third optimization result;

[0034] A deep learning framework reconstruction module is used to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0035] The image result output module is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal, wherein the terminal includes: a memory, a processor, and an image processing program based on a deep learning network model based on edge computing stored on the memory and runnable on the processor, and when the image processing program based on the deep learning network model based on edge computing is executed by the processor, the steps of the image processing method based on the deep learning network model based on edge computing as described above are implemented.

[0037] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program of a deep learning network model based on edge computing, and when the image processing program of a deep learning network model based on edge computing is executed by a processor, the steps of the image processing method of the deep learning network model based on edge computing as described above are implemented.

[0038] In the present invention, the original deep learning network model is obtained, and the original deep learning network model is subjected to vector conversion processing and row vector calculation processing to obtain a first optimization result; the first optimization result is subjected to quantization acceleration processing to obtain a second optimization result; the second optimization result is subjected to indirect buffer construction processing to obtain a third optimization result; the original deep learning network model is subjected to deep learning framework reconstruction processing according to the third optimization result to obtain a target deep neural network model; the target image data is obtained, and the target image data is input into the target deep neural network model, and the target image analysis result is output. The present invention is based on technical processing such as GEMM (general matrix multiplication) vector calculation, CUDA (parallel computing device architecture) quantization programming and TensorRT (deep learning reasoning parameter) tensor reasoning, which effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization in the existing technology of CUDA accelerated deep learning network model in edge computing. At the same time, it realizes the real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices, effectively improving the processing efficiency and accuracy of image data at the edge. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flowchart of a preferred embodiment of the image processing method of the deep learning network model based on edge computing of the present invention;

[0040] Figure 2 This is a schematic diagram of a CUDA accelerated deep learning framework implementation process of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0041] Figure 3 It is a schematic diagram of the communication and calculation methods of edge computing and cloud computing in a preferred embodiment of the image processing method based on the deep learning network model of edge computing of the present invention;

[0042] Figure 4 2 is a schematic diagram of the convolution matrix calculation of im2col in a preferred embodiment of the image processing method of the deep learning network model based on edge computing of the present invention;

[0043] Figure 5 2. It is a schematic diagram showing a comparison of a GEMM algorithm using optimized matrix multiplication in a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0044] Figure 6 Schematic diagram of the difference between GPU and CPU programming in a preferred embodiment of the image processing method based on a deep learning network model of edge computing of the present invention;

[0045] Figure 7 Schematic diagram of matrix multiplication to convolution in a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0046] Figure 8 Schematic diagram of a deep convolution implementation method of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0047] Figure 9 2 is a schematic diagram of an NVIDIA inference stack of a preferred embodiment of an image processing method based on a deep learning network model of edge computing of the present invention;

[0048] Figure 10 This is a structural diagram of a preferred embodiment of an image processing system based on a deep learning network model of edge computing of the present invention;

[0049] Figure 11 FIG. 4 is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] With the rapid development of artificial intelligence and the Internet of Things (IoT) technologies, edge computing devices are playing an increasingly important role in processing real-time, large-scale data. However, edge devices typically have limited computing resources and storage capabilities, making it difficult to meet the high-performance requirements of deep learning models. To address this issue, using CUDA (Compute Unified Device Architecture, a parallel computing platform and programming model, generally referred to as the Unified Device Architecture / Parallel Computing Device Architecture) to accelerate deep learning networks on NVIDIA (NVIDIA Corporation, graphics card chips and motherboard chips) GPUs (graphics processing units) has become an effective technical approach. CUDA is a parallel computing platform and programming model launched by NVIDIA that allows developers to use GPUs for efficient parallel computing general processing, including accelerated computing of deep learning tasks.

[0052] The two core technologies for accelerating deep learning with CUDA programming on edge devices are CUDA programming and TensorRT reasoning. 1. Through CUDA programming, developers can directly access the hardware resources of the GPU and use the parallel computing capabilities of the GPU to accelerate the training and reasoning process of deep learning models. Currently, software frameworks for building and training deep learning models, including mainstream deep learning frameworks such as TensorFlow, PyTorch, and Keras, all support CUDA acceleration. 2. NVIDIA's high-performance deep learning inference optimizer TensorRT (a software development kit for optimizing trained deep learning models for high-performance reasoning) can further accelerate the reasoning process of deep learning models. TensorRT supports importing models from multiple deep learning frameworks and provides a series of optimization technologies, such as layer fusion, dynamic memory management, etc., to improve reasoning speed and efficiency.

[0053] Existing technologies have the following shortcomings: 1. High learning cost: CUDA technology primarily relies on NVIDIA GPU hardware. CUDA programming requires high developer expertise, including knowledge of parallel computing and GPU architecture. This high learning cost prevents beginners from directly utilizing CUDA programming for acceleration. 2. Model optimization challenges: Optimizing deep learning models for deployment on edge devices is challenging. Techniques such as model quantization, pruning, and compression are used to reduce model size and computational requirements, but these techniques can sometimes lead to performance or accuracy degradation. Furthermore, issues such as version control, model updates, and monitoring need to be effectively addressed. 3. Resource limitations on edge devices: While CUDA technology enables efficient parallel computing on GPUs, edge devices typically have limited computing resources and storage capacity, making them unable to meet the demands of real-time and large-scale data processing. This results in certain limitations when running CUDA-based deep learning models on edge devices, which can lead to reduced accuracy or performance. 4. Network latency and data security: In edge computing environments, devices often need to communicate over a network, making it crucial to ensure the security and privacy of this data during inference. At the same time, if a large amount of data needs to be transmitted between the cloud and edge devices, this may introduce network latency and communication overhead, affecting the real-time performance and accuracy of deep learning models and leading to increased bandwidth and storage costs.

[0054] In summary, while the CUDA-accelerated deep learning network model framework based on edge computing devices has significant technical advantages, it also has some shortcomings and challenges. Future research and development should focus on reducing learning costs, improving model optimization performance, optimizing edge device resources, and reducing network latency and data security to promote the widespread application and development of deep learning technologies based on edge computing devices.

[0055] To address these issues, the present invention provides a CUDA acceleration framework and process for edge-side deep learning. The framework specifically includes the following steps: 1. GEMM (General Matrix Multiplication) is used to implement matrix multiplication, which has its own set of rules and strategies for matrix multiplication (matrix multiplication is element-by-element multiplication) and im2col (image to column, sliding a template by column, then converting the data contained in each window into a column vector, and finally arranging it into a new matrix by column); 2. GEMM optimization strategy; 3. QNNPACK (Quantized Neural Network Package) quantization processing strategy; 4. Creating an indirect buffer to implement convolution operations; 5. TensorRT (high-performance deep learning inference optimizer) tensor conversion; and 6. CUDA programming implementation. This invention realizes the real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices. Based on technical processing such as GEMM vector calculation, CUDA quantization programming and TensorRT tensor reasoning, it effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization of existing edge computing CUDA accelerated deep learning network models.

[0056] The image processing method based on the deep learning network model of edge computing described in the preferred embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown, the image processing method based on the deep learning network model of edge computing includes the following steps:

[0057] Step S10: Obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result.

[0058] GEMM matrix multiplication and im2col (image to column) vectorization: Leveraging the advantages of GEMM matrix computation (the basic GEMM matrix method is: 1. Divide the output into several 4×4 sub-blocks to improve input data reuse while making extensive use of registers and reducing memory accesses; 2. Vectorize memory access and computation; 3. Eliminate pointer calculations; 4. Reorganize memory for contiguous addresses), and referencing im2col's implementation of converting 3D input data into a 2D matrix (that is, converting the image's length, width, and number of channels into 2D), we reimplement the main im2col algorithm logic for GPU programming in CUDA. (Using GPU programming strategies and ideas, we rewrite the im2col algorithm, including the CPU matrix multiplication logic and corresponding function methods.) im2col flattens the input data to fit the filter (weights). Specifically, for the input data, the region where the filter is applied (the 3D block) is horizontally flattened into a single column. im2col performs this flattening process everywhere the filter is applied.

[0059] GEMM optimization strategy: Using row vector calculations (calculating row by row, rather than column by column or element by matrix), it replaces and optimizes matrix multiplication in registers, reduces access to data in non-contiguous memory areas, and calculates logical operations (such as addition, subtraction, multiplication, and division) of a 4*4 matrix at one time, optimizing the forward and backward feedback networks and the total inference time of the neural network model.

[0060] Specifically, an original deep learning network model and target three-dimensional data are obtained, and the target three-dimensional data are input into the original deep learning network model; an image vectorization function is obtained, and the image vectorization function is rewritten using parallel computing device architecture programming to obtain a rewritten image vectorization function; vector conversion processing is performed on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix; a general matrix multiplication optimization strategy is used to perform row vector calculation processing on the target two-dimensional matrix to obtain a first optimization result, wherein the general matrix multiplication optimization strategy includes address offset processing, simplified loop processing, and memory alignment processing.

[0061] like Figure 3 As shown, edge devices perform computations locally, closer to the data source or the “edge” of the network rather than relying solely on centralized cloud servers. Figure 3 This includes smartphones, IoT devices, and edge servers. Figure 3 As shown in the example, Figure 3A in the figure connects the drone (perception terminal), edge computing device (computing terminal) and 4G / 5G communication base station (communication terminal). Figure 3 The B in the image connects drones, cloud servers, and communication base stations. By leveraging CUDA acceleration on edge devices, deep learning inference tasks can be performed faster and more efficiently, enabling real-time or near-real-time data processing without relying on remote servers. This integration allows tasks such as image recognition, natural language processing, and sensor data analysis to be performed locally, which is critical for applications with low latency or limited connectivity.

[0062] GEMM matrix multiplication and im2col vector conversion: In the original CUDNN (CUDA Deep Neural Network, a GPU-accelerated library for deep neural networks) framework, NVIDIA integrated deep learning matrix computations without providing layer abstractions. Instead, it provided lower-level computational primitives to simplify integration with existing deep learning frameworks. These included the classic Fourier transform (FFT), Volker-Strassen, and Winograd. 1. The Fourier transform and fast Fourier transform are commonly used in classical image processing, but are not typically employed in ConvNets. This is primarily because the convolution masks in ConvNets are typically small, such as 3×3. In this case, the FFT is more expensive, making it rarely used for convolution in CNNs. 2. The Volker-Strassen method only shows significant advantages when using large convolution kernels. Currently, this method has not been seen in open source frameworks, due to kernel size limitations. 3. In most scenarios, the Winograd method demonstrates significant advantages and is currently used for convolution in CUDNN.

[0063] Current neural networks used for computer vision spend the majority of their inference time on convolution and fully connected operators. These operators are closely related to matrix multiplication: fully connected operators and 1×1 convolutions map directly to matrix multiplications, and convolutions with larger kernels can be decomposed into a combination of a memory layout transformation called im2col and matrix multiplications. Therefore, the problem of efficient inference in convolutional neural networks is largely a matter of efficiently implementing matrix multiplication, also known as GEMM in linear algebra libraries. With the adoption of the im2col + GEMM combination in the Caffe (Convolutional Architecture for Fast Feature Embedding) framework, nearly all mainstream computing frameworks, including Caffe, MXNet, and TensorFlow, have implemented this approach. This approach transforms the entire convolution process into a GEMM, which is highly optimized in various BLAS libraries and generally offers high speed.

[0064] like Figure 4 As shown, im2col converts the input 3D data into a 2D matrix, which allows the convolution operation to be represented as the multiplication of two 2D matrices, thereby fully utilizing the optimized GEMM library to accelerate convolution operations on various platforms. The cost of using GEMM for convolution calculations in im2col is the additional memory overhead, using an additional x times the memory for the input.

[0065] Among them, CUDA programming is used to rewrite im2col, as shown in the following function body:

[0066]

[0067]

[0068] GEMM optimization strategy: To solve the above problems, its basic method is to divide the output (the output is a two-dimensional matrix, which refers to the output of Im2col, that is, the result of multiplying the input image and the convolution kernel. The storage format and representation are the same as the original im2col, except that the logic code is rewritten using CUDA) into several 4×4 sub-blocks. When these elements are single-precision floating-point numbers, the memory size is 16 bytes, which can improve the reuse of input data. At the same time, registers are used extensively to reduce memory access. The purpose is to directly load 4 elements instead of using 4 discontinuous memory accesses 4 times; vectorized memory access and calculation. During the CUDA vector calculation process, row vector calculation is used to implement 4 vector accesses loaded into registers to realize the calculation of the original 16 elements; pointer calculation is eliminated, and reference calculation is used instead for the pointer of input matrix B; memory is reorganized to have continuous addresses, etc.

[0069] Now we calculate a 4x 4 block of C at a time (a row of A multiplied by a column of B to get an element of C, where A is the input, B is the weight, and C is the output) in order to use vector instructions and vector registers efficiently. The idea is as follows: There are special instructions in the SSE3 instruction set that allow two "multiplication-accumulation" operations (two multiplications and two additions) to be performed per clock cycle, for a total of four floating-point operations per clock cycle. To use these, the data must be placed in "vector registers." There are sixteen of them, each of which can hold two double-precision numbers. Therefore, 32 double-precision numbers can be kept in the register. Sixteen of them will be used to hold the elements of C, that is, 4x 4 blocks. Among them, the horizontal axis is the matrix size. For easier operation, the MNK value is assumed to be the same, and the vertical axis is gflops. The original version of the matrix multiplication has less than 1 gflops, and it will decrease as the matrix size increases, such as Figure 5 As shown (representing the number of operations Gflops before and after GEMM optimization).

[0070] Its core optimization strategy includes the following three aspects:

[0071] 1. Avoid multiplication:

[0072] for(p=0;p <k;p++){

[0073] a_0p_reg=A(0,p);

[0074] c_00_reg+=a_0p_reg*B(p, 0);

[0075] c_01_reg+=a_0p_reg*B(p, 1);

[0076] c_02_reg+=a_0p_reg*B(p, 2);

[0077] c_03_reg+=a_0p_reg*B(p, 3);

[0078] }

[0079] Each calculation of B(p, 0) requires a multiplication (B(p, 0) represents the element position of matrix B, row p, column 0). We can use a pointer to store the address of B(p, 0) and offset it when B(p, 1) is needed. For example, the original text is in column-major order, and the modified code snippet is as follows:

[0080] for(p=0;p <k;p++){

[0081] a_0p_reg=A(0,p);

[0082] c_00_reg+=a_0p_reg**bp0_pntr++;

[0083] c_01_reg+=a_0p_reg**bp1_pntr++;

[0084] c_02_reg+=a_0p_reg**bp2_pntr++;

[0085] c_03_reg+=a_0p_reg**bp3_pntr++;

[0086] }

[0087] Therefore, 4 multiplications are saved (the original code required multiplication and addition each time, but now it uses pointer reference to directly reference the storage address. It is necessary to take the value each time, a total of 4*4 area, saving 4 multiplication operations).

[0088] 2. Simplify the loop: Simple loop structures are more conducive to compiler optimization.

[0089] 3. Memory alignment: Assume that the cache line size is 32B. If the data to be accessed is 64B in size and the address is 0x80000001, then three cache mapping entries are required. If the address is 0x80000000, only two entries are required. Memory alignment indirectly improves the cache hit rate.

[0090] Taking the classification deep learning network AlexNet (input size 128x3x224x224) as an example, the comparison of its optimized front and back feedback networks and total inference time is shown in Table 1:

[0091] Table 1: Comparison of inference speed of each framework of AlexNet network

[0092] Library Time (ms) forward(ms) backward(ms) CuDNN-fp16 71 25 46 CuDNN-fp32 81 27 53 CuDNN+Caffe 231 70 161 Caffe 324 121 203 Torch-7 342 132 210

[0093] By leveraging the respective advantages of GEMM matrix multiplication and im2col vector calculation, and reconstructing convolution operations and accumulation calculations using row vector GPU programming logic, we can achieve fast calculation of the underlying operation logic (addition, subtraction, multiplication, division, etc.) and solve the memory usage problem.

[0094] Step S20: Perform quantization acceleration processing on the first optimization result to obtain a second optimization result.

[0095] The QNNPACK quantization processing strategy utilizes the PDOT microkernel primitive (panel dot product, a matrix multiplication method where primitives are units) to implement high-performance convolution-matrix multiplication. Quantized GPU (CUDA) programming is used to implement memory copy logic, enabling multiplication and accumulation while alleviating the limitations of memory and cache bandwidth on convolutional layer operations. (CUDA computing is GPU computing, and the GPU has its own computing logic and implementation framework. The CPU logic code can be rewritten based on this framework.)

[0096] While it's possible to implement the entire matrix multiplication on the dot product primitive, this implementation is inefficient. In a dot product, each multiply-add operation requires uploading two elements. On current processors, this implementation is limited by memory and cache bandwidth, not the computational power of the multiply-add unit. However, a small modification—computing the dot products of several rows of A and B simultaneously—significantly improves performance.

[0097] The modified primitive loads the MR element of A and the NR element of B, performing the MRxNR multiply-accumulate operation. The maximum values ​​of MR and NR are limited by the number of integers and other details of the processor architecture. However, on most modern systems, these maximum values ​​are large enough to make the operation limited only by computation. All high-performance matrix multiplication implementations are built on this primitive, which is often referred to as the PDOT (panel dot product) microkernel. (In other words, multiplication used to require individual computations, which would be limited by memory and cache because each computation required space to store one element. Now, by implementing the computation in the primitive, the computation can be performed directly and the results output to memory and cache without the need for intermediate storage.)

[0098] Specifically, a quantized neural network package is obtained, and a microkernel primitive is constructed based on the quantized neural network package; high-performance matrix multiplication is used to perform quantized acceleration processing on the first optimization result based on the microkernel primitive to obtain a second optimization result.

[0099] QNNPACK quantization processing strategy: Caffe2Go uses a kernel library called NNPACK, which implements a progressively fast convolution algorithm based on Winograd transform or fast Fourier transform, allowing convolution calculations to use several times fewer multiplication operations than direct algorithms. For example, a 3×3 convolution can only be 2 times slower than a 1×1 convolution, while the direct algorithm is 9 times slower.

[0100] QNNPACK (Quantized Neural Network Package) is part of Facebook's application suite and has been deployed on billions of mobile devices worldwide. This new library, QNNPACK, can perform advanced computer vision tasks, such as running Mask R-CNN (an AI object detection algorithm) and DensePose (an AI pose estimation algorithm) in real time on mobile phones, or performing image classification in under 100 milliseconds on performance-constrained mobile devices.

[0101] For example, A is the input, B is the weight, and C is the output. During inference, B never changes and can be migrated to any convenient storage configuration without wasting time. The MxK matrix A is multiplied by the KxN matrix B to produce the MxN matrix C, where each element in C can be thought of as the dot product of the row of A and the corresponding column of B.

[0102] It's possible to implement the entire matrix multiplication on the dot product primitive, but such an implementation is far too inefficient. In a dot product, each multiply-add operation requires uploading two elements. On current processors, this implementation is limited by memory and cache bandwidth, not the computational power of the multiply-add unit. However, a small modification to compute the dot products of several rows of A and B simultaneously yields a significant performance boost.

[0103] like Figure 6 As shown in the figure, by comparing CPU programming and GPU (CUDA) programming, we can see the difference in the implementation of memory copy logic.

[0104] This modified primitive loads the MR (a block of computational units or storage cells with M rows and R columns, often referred to as the MR matrix) of A and the NR (N rows and R columns) elements of B, performing the MRxNR multiply-accumulate operation. The maximum values ​​of MR and NR are limited by the number of integers and other details of the processor architecture. However, on most modern systems, these maximum values ​​are large enough to limit the operation to computational constraints. All high-performance matrix multiplication implementations are built on this primitive, often referred to as the PDOT (panel dot product) microkernel.

[0105] QNNPACK uses a linear quantization scheme compatible with the Android Neural Network API. It assumes that the quantized values ​​q[i] are represented as 8-bit unsigned integers and are related to the real-valued representation r[i] by the following formula: r[i] = scale * (q[i] - zero_point); where scale is a positive floating-point number and zero_point is an unsigned 8-bit integer, the same as q[i]. Using the QNNPACK quantization strategy further reduces the computational complexity of the network model and the resource consumption of floating-point calculations, making it possible to apply deep learning network models to edge computing devices while maintaining network accuracy.

[0106] Step S30: Perform indirect buffer construction processing on the second optimization result to obtain a third optimization result.

[0107] Creating an indirect buffer for convolution operations: Leveraging the advantages of QNNPACK indirect buffers, we avoid the increased memory overhead introduced by im2col vectorization. Furthermore, during spatial convolution, we use indirect pointers to fixed addresses where zeros are padded, eliminating the complexity of memory address pointing calculations.

[0108] Specifically, the input buffer of the original deep learning network model is obtained, and an indirect buffer is constructed based on the input buffer using an indirect convolution algorithm; and a pointer simulation method is used to perform memory access simulation processing on the second optimization result based on the indirect buffer to obtain a third optimization result.

[0109] like Figure 7 As shown, Figure 7 Detailed process of the indirect convolution algorithm workflow. Figure 7 The a in indicates that multiple inputs use the same input buffer. The indirect convolution algorithm builds the following Figure 7 b in Figure 7 The b in is the [Indirect Buffer], and the indirect buffer is the core of the indirect convolution algorithm. Figure 7 In the b, when the network runs, each time it computes an M×N (O rows and H columns) output, where M is the vectorized size after treating OH×OW as one dimension. Typically, M×N is 4×4, 8×8, or 4×8. When computing an M×N output, the corresponding input is retrieved from the indirect buffer, along with the weights, to calculate the result. This computation is equivalent to performing an M×K matrix multiplication with a K×N matrix.

[0110] In the specific implementation process, the software execution process is divided into two parts:

[0111] 1. Preparation phase: load the model, configure the input buffer; rearrange the weights so that its memory layout is suitable for subsequent calculations; 2. Run phase: for each input, run [OH*OW / M][OC / N] core loops, each time using the GEMM method to calculate an M×N output.

[0112] The indirect buffer allows the simulation of memory access to the input through pointers. When the micro kernel that calculates the M×N output is actually running, there will be M pointers scanning the input (image or matrix, output of the previous layer).

[0113] The Im2col optimization algorithm first copies the input into a matrix, such as Figure 8 The indirect buffer used by the indirect convolution optimization algorithm actually stores pointers to the input (this is why the indirect convolution optimization algorithm requires the input memory address to be fixed). At runtime, these pointers can be used to simulate a matrix calculation process similar to Im2col.

[0114] The indirect convolution optimization algorithm solves three problems in convolution calculations: the first is spatial vectorization, the second is complex address calculation, and the third is memory copying. Generally, when calculating convolution, the input needs to be padded with zeros (for cases where KH×KW is not 1×1). Traditional methods in this process will result in memory copies. However, the indirect buffer in the indirect convolution optimization algorithm cleverly solves this problem through indirect pointers. When constructing the indirect buffer, an additional 1×IC (I-row, C-column matrix) memory buffer is created and filled with zero values. For locations in the space that need to be padded with zeros, the corresponding indirect pointer points to this buffer, so that subsequent calculations are equivalent to having already padded with zeros.

[0115] In QNNPACK, the present invention implements a more efficient algorithm that does not transform the convolution input to accommodate the matrix multiplication implementation. Instead, the implementation of the PDOT microkernel is adjusted to perform the im2col transformation virtually. Instead of copying the actual data from the input tensor to the im2col buffer, an indirect buffer is set up with pointers to the rows of input pixels involved in computing each output pixel. The matrix multiplication microkernel is also modified (load the pointer (matrix) to be calculated from the indirect buffer) to load pointers to the rows of the imaginary matrix A from the indirect buffer, which is typically much smaller than the im2col buffer. Furthermore, if the memory location of the input tensor does not change between inference runs, the indirect buffer can also be initialized once with pointers to the input rows and then reused across multiple inference runs. It can be observed that the microkernel with the indirect buffer not only eliminates the overhead of the im2col transformation, but also performs slightly better than the matrix multiplication microkernel (probably because the input rows are reused when computing different output pixels).

[0116] like Figure 8 As shown in the quantized MobileNetV2 benchmark, QNNPACK outperforms state-of-the-art implementations by approximately 2x on various mobile phones. The main change is in the convolution. For a 3×3 depthwise convolution, each output row is updated nine times. QNNPACK calculates the results of all 3×3 convolution kernel rows and 3×3 input rows, accumulates them in one output row, and then processes the next output row.

[0117] Step S40: Perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model.

[0118] TensorRT tensor conversion: Based on NVIDIA's native deep learning framework, it combines the key features of the TensorRT inference engine on the edge to implement matrix vectorization operations and complete inference reconstruction of each layer of the model based on factors such as its workspace size, input image matrix format, and quantized floating-point operations.

[0119] CUDA programming implementation: Based on the three characteristics of deep learning of the cuDNN acceleration library, and based on GEMM vectorized calculations, QNNPACK quantization acceleration, and TensorRT tensor conversion, the deep learning framework of CUDA programming (cuDNN acceleration library) is reconstructed.

[0120] Specifically, deep learning inference parameters are obtained, and matrix vectorization processing is performed on the third optimization result according to the deep learning inference parameters to obtain a vectorized result, wherein the deep learning inference parameters include workspace size, input image matrix format, and quantized floating-point operations; deep learning framework reconstruction processing is performed on the vectorized result according to the parallel computing device architecture programming to obtain a target deep neural network framework.

[0121] TensorRT tensor conversion: TensorRT can allocate workspace memory for intermediate buffers inside the network. When it comes to execution, TensorRT can generally provide faster execution because it guarantees to choose the best execution path for the entire graph rather than just a sub-part. This obviously comes at the cost of such high engine creation time, so some deployment issues need to be considered, such as Figure 9 As shown, Figure 9 Three important parameters for deploying TensorRT for CUDA: 1. TensorRT workspace size; 2. NHWC format (N (samples) dimensions H height W width C number of channels); 3. FP16 and FP8 (16-bit floating point and 8-bit floating point).

[0122] The workspace size of TensorRT is an important parameter of TensorRT. Since TensorRT can rearrange the operations in the graph for optimization, it may need more memory to store intermediate results. This value is controlled by the workspace size, which tells the engine builder how many bytes TensorRT can allocate in addition to the engine weight and input. It makes sense to experiment with this parameter and trade off speed with inference size.

[0123] NHWC format inputs are well-suited for Tensor Core GPUs on NVIDIA. Since ONNX only supports the NCHW format, a trick must be used to enable NHWC as an input tensor. Set the input dimensions to NHWC and insert a Transpose operation after the input removed by CUDA or TensorRT EP.

[0124] Operation precisions such as FP16 and FP8 are particularly important for optimal performance on GPUs. In TensorRT EP, they must be explicitly enabled during session creation using the following property: OrtTensorRTProviderOptions trt_options{};

[0125] trt_options.trt_fp16_enable=1;

[0126] trt_options.trt_int8_enable=1;

[0127] CUDA programming implementation: Currently, NVIDIA provides the cuDNN acceleration library to accelerate the operation of deep learning networks, which mainly includes: 1. Accelerated learning; 2. Expression operation graph API; 3. Fusion support.

[0128] 1. Accelerated learning: cuDNN provides kernels that target Tensor cores to provide the best available performance on compute-bound operations. It provides heuristics for selecting the right kernel for a given problem size.

[0129] 2. Expression Graph API: Users define computations as graphs of operations on tensors. The cuDNN library has both a direct C API and an open-source C++ front-end for ease of use, with most users choosing the front-end as their entry point to cuDNN.

[0130] 3. Fusion support: cuDNN supports the fusion of compute-bound and memory-bound operations. Common general-purpose fusion modes are typically implemented through runtime kernel generation. Specialized fusion modes are optimized using pre-written kernels.

[0131] CuDNN benchmarks were done using Torch bindings, but the same can be done with Caffe bindings or bindings for any other library. This disclosure aims to clarify that Caffe (native) and Torch (native) are the convolution kernels that appear as the default fallback. Some frameworks, such as TensorFlow and Chainer, are benchmarked against CuDNN without explicitly mentioning this, so one might assume that these frameworks as a whole are faster than Caffe, which might not be the case.

[0132] Based on GEMM vectorized calculation, QNNPACK quantization acceleration and TensorRT tensor conversion, this paper realizes the reconstruction of the deep learning framework of CUDA programming (cuDNN acceleration library), including the rewriting of Conv, flatten, max_pool and other layer networks (Cuda implementation).

[0133] Take forward as an example

[0134]

[0135] Table 2: Edge intelligence device environment configuration

[0136]

[0137] This invention is based on the features of the CUDNN acceleration library, combines the advantages of the TensorRT reasoner for tensor computing, and uses CUDA programming to rebuild the implementation logic of each layer of the network model, which can more quickly realize the rapid transplantation and lightweighting of different deep learning frameworks in edge devices.

[0138] Step S50: Acquire target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0139] Specifically, target image data is acquired and input into the target deep neural network model; image analysis processing is performed on the target image data through the target deep neural network model, and a target image analysis result is output.

[0140] After accelerating the neural network model deployed at the edge, a target deep neural network model is generated. This accelerated target deep neural network model can parse and process the input target image data without being limited by the computing resources and storage capacity of the edge device. This accelerated target deep neural network model also improves image processing efficiency and accuracy at the edge.

[0141] In addition, the input of the target deep neural network model in the present invention is not limited to image recognition, but can also be natural language processing data or sensor data, all of which can be efficiently processed at the edge through the target deep neural network model.

[0142] In summary, the present invention is based on the CUDA accelerated deep learning network model framework of edge computing devices, and relates to an acceleration method for the edge-side deep learning network framework in the fields of smart cities and low-altitude economy. First, based on the framework acceleration principle of GEMM matrix multiplication and the efficient computing characteristics of im2col vectorized calculation, row calculation is used to replace matrix calculation to reduce the problem of memory occupancy. Secondly, the QNNPACK quantization processing strategy is adopted to optimize the problem of discontinuous memory pointing in row calculations, and the strategy of indirect cache specific pointer pointing is adopted to realize the replacement of deep convolution operation pairs. Finally, the TensorRT reasoner and cuDNN acceleration library are used, and CUDA programming is used to reconstruct the logic of data processing at each layer of the neural network to realize the application of deep learning models of different frameworks in edge devices. The present invention realizes the real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices, and effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization of existing edge computing CUDA accelerated deep learning network models.

[0143] Furthermore, if Figure 10As shown, based on the above-mentioned image processing method of the deep learning network model based on edge computing, the present invention also provides an image processing system based on the deep learning network model based on edge computing, wherein the image processing system based on the deep learning network model based on edge computing includes:

[0144] A first optimization processing module 51 is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result;

[0145] A second optimization processing module 52 is used to perform quantization acceleration processing on the first optimization result to obtain a second optimization result;

[0146] A third optimization processing module 53 is configured to perform an indirect buffer construction process on the second optimization result to obtain a third optimization result;

[0147] A deep learning framework reconstruction module 54 is configured to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model;

[0148] The image result output module 55 is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

[0149] Furthermore, if Figure 11 As shown, based on the above-mentioned image processing method and system of the deep learning network model based on edge computing, the present invention also provides a terminal accordingly, which includes a processor 10, a memory 20 and a display 30. Figure 11 Only some of the components of the terminal are shown, but it should be understood that implementation of all of the shown components is not required, and more or fewer components may be implemented instead.

[0150] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Furthermore, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as the program code of the installation terminal. The memory 20 can also be used to temporarily store data that has been output or is to be output. In one embodiment, an image processing program 40 based on a deep learning network model of edge computing is stored on the memory 20, and the image processing program 40 based on a deep learning network model of edge computing can be executed by the processor 10, thereby realizing the image processing method based on a deep learning network model of edge computing in the present application.

[0151] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing the image processing method based on the edge computing deep learning network model.

[0152] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0153] In one embodiment, when the processor 10 executes the image processing program 40 based on the deep learning network model based on edge computing in the memory 20, the steps of the image processing method based on the deep learning network model based on edge computing as described above are implemented.

[0154] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores an image processing program of a deep learning network model based on edge computing, and when the image processing program of a deep learning network model based on edge computing is executed by a processor, the steps of the image processing method of the deep learning network model based on edge computing as described above are implemented.

[0155] In summary, the present invention provides an image processing method and related equipment based on a deep learning network model of edge computing, the method comprising: obtaining an original deep learning network model, performing vector conversion processing and row vector calculation processing on the original deep learning network model to obtain a first optimization result; performing quantization acceleration processing on the first optimization result to obtain a second optimization result; performing indirect buffer construction processing on the second optimization result to obtain a third optimization result; performing deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; obtaining target image data, and inputting the target image data into the target deep neural network model to output a target image analysis result. The present invention is based on technical processing such as GEMM (general matrix multiplication) vector calculation, CUDA (parallel computing device architecture) quantization programming and TensorRT (deep learning reasoning parameter) tensor reasoning, which effectively solves the problems of high learning cost, low computing performance of edge devices and difficulty in model optimization in the existing technology of CUDA acceleration of deep learning network models in edge computing. At the same time, it realizes real-time calculation, fast and efficient learning and reasoning tasks of deep learning models in edge computing devices, effectively improving the processing efficiency and accuracy of image data at the edge.

[0156] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or terminal comprising the element.

[0157] Of course, those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium that can be read by a computer. When the program is executed, it can include the processes in the above-described method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disk, etc.

[0158] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An image processing method based on a deep learning network model of edge computing, characterized in that: The image processing method based on the deep learning network model of edge computing includes: Acquire an original deep learning network model and target three-dimensional data, and input the target three-dimensional data into the original deep learning network model; Obtaining an image vectorization function, and rewriting the image vectorization function using parallel computing device architecture programming, using GPU programming strategies and concepts, including matrix multiplication logic and corresponding function methods, to obtain the rewritten image vectorization function; Performing vector conversion processing on the target three-dimensional data input into the original deep learning network model according to the rewritten image vectorization function to obtain a target two-dimensional matrix; Performing row vector calculation processing on the target two-dimensional matrix using a general matrix multiplication optimization strategy to obtain a first optimization result; Obtaining a quantized neural network package, and constructing a microkernel primitive based on the quantized neural network package, adjusting the implementation of the microkernel primitive in the quantized neural network package, and virtually performing an image vectorization transformation; Performing quantized acceleration processing on the first optimization result according to the microkernel primitive using high-performance matrix multiplication to obtain a second optimization result; Obtaining an input buffer of the original deep learning network model, and constructing an indirect buffer based on the input buffer using an indirect convolution algorithm; Using a pointer simulation method to perform memory access simulation processing on the second optimization result according to the indirect buffer to obtain a third optimization result; In the spatial zero-padding position during spatial convolution, an indirect pointer is used to point to the spatial zero-padding position to solve the problem of complex memory address pointing calculation; Obtaining deep learning inference parameters, and performing matrix vectorization processing on the third optimization result according to the deep learning inference parameters to obtain a vectorized result, wherein the deep learning inference parameters include a workspace size, an input image matrix format, and a quantized floating-point operation; Performing deep learning framework reconstruction processing on the vectorization result according to the parallel computing device architecture programming to obtain a target deep neural network framework; Obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

2. The image processing method based on the deep learning network model of edge computing according to claim 1 is characterized in that The acquiring target image data, inputting the target image data into the target deep neural network model, and outputting the target image analysis result specifically includes: Acquire target image data, and input the target image data into the target deep neural network model; The target image data is subjected to image analysis processing by the target deep neural network model, and a target image analysis result is output.

3. The image processing method based on the deep learning network model of edge computing according to claim 1, characterized in that The general matrix multiplication optimization strategy includes address offset processing, loop simplification processing and memory alignment processing.

4. An image processing system based on a deep learning network model of edge computing, characterized in that: The image processing system based on the deep learning network model of edge computing is applied to the image processing method based on the deep learning network model of edge computing according to any one of claims 1 to 3, and the image processing system based on the deep learning network model of edge computing includes: A first optimization processing module is used to obtain an original deep learning network model, perform vector conversion processing and row vector calculation processing on the original deep learning network model, and obtain a first optimization result; A second optimization processing module is used to perform quantization acceleration processing on the first optimization result to obtain a second optimization result; A third optimization processing module, configured to perform indirect buffer construction processing on the second optimization result to obtain a third optimization result; A deep learning framework reconstruction module is used to perform deep learning framework reconstruction processing on the original deep learning network model according to the third optimization result to obtain a target deep neural network model; The image result output module is used to obtain target image data, input the target image data into the target deep neural network model, and output the target image analysis result.

5. A terminal, characterized in that: The terminal includes: a memory, a processor, and an image processing program based on a deep learning network model based on edge computing, which is stored in the memory and can be run on the processor. When the image processing program based on the deep learning network model based on edge computing is executed by the processor, the steps of the image processing method based on the deep learning network model based on edge computing are implemented.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium stores an image processing program of a deep learning network model based on edge computing. When the image processing program of the deep learning network model based on edge computing is executed by a processor, the steps of the image processing method of the deep learning network model based on edge computing are implemented as described in any one of claims 1 to 3.

Images