Intelligent terminal CPU-GPU parallel inference method

Abstract

The application discloses an intelligent terminal CPU-GPU parallel reasoning method, characterized by adopting a local filling and division boundary perception method to realize delay-accuracy balanced CPU-GPU parallel reasoning, and the method specifically comprises the following steps: determining an optimal division mode according to the edge features of a division boundary for an input picture, respectively giving each image partition to a CPU and a GPU for parallel reasoning of a neural network, and finally merging before an output layer of the neural network, wherein necessary pseudo data is locally filled before some layers of the neural network. Compared with the prior art, the application has the advantages that the data sharing overhead of CPU-GPU parallel reasoning is greatly reduced through local filling of pseudo data, and reasoning delay is significantly reduced. In addition, by using the division method of division boundary perception, delay-accuracy balanced CPU-GPU parallel reasoning is realized, the method is simple, the use effect is good, and the application is not only suitable for intelligent terminals, but also suitable for personal computers and servers, and has a good application prospect.

Description

Technical Field

[0001] This invention relates to the field of neural network inference technology, and in particular to a CPU-GPU parallel inference method for intelligent terminals based on local filling and image segmentation boundary awareness. Background Technology

[0002] With the surge in deep learning-based applications on smart devices, such as photo enhancement, face detection, and object recognition, major internet companies have launched deep learning frameworks specifically for smart devices, such as Tensorflow Lite, MACE, and NCNN. Running the entire neural network on a local device effectively avoids the risks of user privacy leaks and network signal fluctuations, thus on-device inference is rapidly gaining popularity. Generally, current mainstream deep learning frameworks typically use a single processor, such as a GPU, to execute the entire neural network, which misses the opportunity to utilize existing heterogeneous processor resources for acceleration. Furthermore, due to limitations in size and power consumption, mobile GPUs perform worse than traditional servers. Mobile users often complain about the slow response times of CNN-related services. In recent years, some research has attempted to accelerate neural network inference by leveraging the existing heterogeneous processor resources of smart devices. Some studies reduce latency by allocating different layers of the neural network to different processors, but each layer can only be computed on the selected processor, meaning the processing of the neural network is essentially still serial.

[0003] The performance gap between CPUs and GPUs in neural network inference on current smart terminals is not significant, with CPUs playing a more crucial role. Therefore, to further improve performance, some researchers have accelerated each layer of the neural network by coordinating CPU and GPU parallel processing. Different partitions are assigned to the CPU and GPU for parallel processing by dividing the network along the output channel or height dimension. However, processors need to share the output of the previous layer to enable parallel inference of the next layer, as convolution and pooling operations require processing data across image partition boundaries. This cross-processor data sharing significantly increases inference latency, meaning that the performance improvement of CPU-GPU parallel inference on smart terminals still falls short of user expectations, and even accelerates lightweight neural networks. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a CPU-GPU parallel inference method for intelligent terminals. By locally padding necessary pseudo-data on the CPU and GPU, this method ensures that the neural network only needs to be computed locally on the CPU and GPU, eliminating data sharing overhead. The method determines the optimal partitioning pattern of the input image based on the edge features of the partitioning boundaries. Each image partition is then fed separately to the CPU and GPU for parallel inference of the neural network, and finally merged before the output layer of the neural network. Local padding eliminates data sharing overhead in CPU-GPU parallel inference on intelligent terminals, offering convenience and simplicity, significantly reducing data sharing overhead in CPU-GPU parallel inference, and significantly lowering inference latency. Utilizing a partitioning boundary-aware method, it achieves a latency-accuracy balance in CPU-GPU parallel inference. The method is simple, effective, and applicable not only to intelligent terminals but also to personal computers and servers, demonstrating good application prospects and commercial development value.

[0005] The specific technical solution to achieve the objective of this invention is: a CPU-GPU parallel inference method for intelligent terminals, which employs local padding and boundary awareness to achieve latency-accuracy balanced CPU-GPU parallel inference, and the method specifically includes:

[0006] Step 1: Input an image that requires neural network inference;

[0007] Step 2: Based on the actual latency of neural network inference on CPU and GPU, obtain the optimal image segmentation ratio (which can be reciprocals of each other, such as 2:3 and 3:2), and combine it with the segmentation dimension to form the candidate segmentation pattern of the input image;

[0008] Step 3: Based on the edge features extracted by edge detection, determine whether it is the partitioning mode with the fewest edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, it is the optimal partitioning mode and the following steps are executed.

[0009] Step 4: Divide the image according to the optimal partitioning mode selected above, and allocate the different partitions to the CPU and GPU respectively for parallel inference of the neural network;

[0010] Step 5: Based on the operator settings and data partition size of this neural network layer, determine whether pseudo-data needs to be filled. If no filling is needed, proceed directly to Step 7; if filling is needed, execute the following steps.

[0011] Step 6: Based on the operator settings and data partition size of this neural network layer, pad the CPU and GPU partitions with zeros of a certain amount, and then execute the following steps;

[0012] Step 7: Perform CPU-GPU parallel inference for this layer of the neural network;

[0013] Step 8: Determine if the next layer of the neural network is an output layer. If not, return to step 5 and perform parallel inference of the neural network again. If it is, execute the following steps.

[0014] Step 9: Merge the output feature maps of the CPU and GPU, and input them into the CPU or GPU to complete CPU-GPU parallel inference with a balance between latency and accuracy.

[0015] The necessary pseudo-data padding on the CPU and GPU processors is precisely determined based on the operator settings and data partition size of each layer of the neural network. By padding the necessary pseudo-data locally on the CPU and GPU processors, each layer of the neural network is computed in parallel on both CPU and GPU, eliminating the need for data sharing between CPU and GPU. Furthermore, considering that there are hundreds or even thousands of feature maps during neural network inference, no existing padding method is suitable for all feature maps due to the hierarchical nature of feature extraction in convolutional layers. Inappropriate padding methods can lead to high overhead or low accuracy. Therefore, zero padding, widely used in neural networks, is employed, which introduces almost no additional overhead.

[0016] The loss of necessary pseudo-data during local filling only involves information across the partition boundaries. Considering the potential presence of numerous redundant pixels in the image that are irrelevant to accurate recognition, we can use intelligent partitioning to minimize the impact of lost information on accurate image recognition. To avoid information loss in important feature regions, this invention selects the optimal partitioning pattern with the fewest edge pixels within a certain interval near the candidate partition boundaries, based on the edge features extracted by edge detection. After partitioning according to the optimal pattern, the different partitions of the input image are then fed into the CPU and GPU for parallel inference of the neural network.

[0017] Compared with existing technologies, this invention significantly reduces data sharing overhead and inference latency by locally filling in pseudo-data for CPU-GPU parallel inference. Furthermore, it achieves latency-accuracy balanced CPU-GPU parallel inference using a boundary-aware partitioning method. The method is simple, effective, and applicable not only to smart terminals but also to personal computers and servers, demonstrating promising application prospects and commercial development value. Attached Figure Description

[0018] Figure 1 is a flowchart of the present invention;

[0019] Figure 2 is a schematic diagram of the candidate partitioning pattern. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0021] Example 1

[0022] The optimal segmentation mode for the input image is determined based on the edge features of the segmentation boundaries. Each image partition is then fed into the CPU and GPU for parallel inference in the neural network, and finally merged before the output layer of the neural network. Necessary pseudo-data is locally padded before certain layers of the neural network to achieve a latency-accuracy balance in CPU-GPU parallel inference. The specific steps are as follows:

[0023] Step 1: Input an image that requires neural network inference;

[0024] Step 2: Based on the actual latency of neural network inference on CPU and GPU, obtain the optimal image segmentation ratio (which can be reciprocals of each other, such as 2:3 and 3:2), and combine it with the segmentation dimension to form the candidate segmentation pattern of the input image;

[0025] Step 3: Based on the edge features extracted by edge detection, determine whether it is the partitioning mode with the fewest edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, it is the optimal partitioning mode and the following steps are executed.

[0026] Step 4: Divide the image according to the optimal partitioning mode selected above, and allocate different partitions to the CPU and GPU respectively for parallel inference of the neural network;

[0027] Step 5: Based on the operator settings and data partition size of this neural network layer, determine whether pseudo-data needs to be filled. If no filling is needed, proceed directly to Step 7; if filling is needed, execute the following steps.

[0028] Step 6: Based on the operator settings and data partition size of this neural network layer, pad the CPU and GPU partitions with zeros of a certain amount, and then execute the following steps;

[0029] Step 7: Perform CPU-GPU parallel inference for this layer of the neural network;

[0030] Step 8: Determine if the next layer of the neural network is an output layer. If not, return to step 5 and perform parallel inference of the neural network again. If it is, execute the following steps.

[0031] Step 9: Merge the output feature maps of the CPU and GPU, and input them into the CPU or GPU to complete CPU-GPU parallel inference with a balance between latency and accuracy.

[0032] The pseudo-data filling in step five is determined by the operator settings and data partition size of each layer of the neural network. The pseudo-data is filled locally by the processor, so that each layer of the neural network can be executed in parallel locally on the CPU and GPU without the need for data sharing between the CPU and GPU.

[0033] See Figure 1 The process of achieving a balance between latency and accuracy in CPU-GPU parallel inference begins in step 201 and then proceeds to step 202.

[0034] In step 202, input an image that requires neural network inference, and then execute step 203.

[0035] See Figure 2 In step 203, the optimal image segmentation ratio (which can be reciprocals of each other, such as 2:3 and 3:2) is obtained based on the actual latency of the neural network in CPU and GPU inference, and combined with the segmentation dimension to form the candidate segmentation pattern of the input image, and then step 204 is executed.

[0036] In step 204, based on the edge features extracted by edge detection, it is determined whether this is the optimal partitioning mode with the fewest edge pixels within a certain interval near the candidate partitioning boundary. If not, this partitioning mode is discarded, and the process proceeds directly to step 212. If yes, step 205 is executed.

[0037] In step 205, the image is divided according to the selected optimal division mode, and then step 206 is executed.

[0038] In step 206, the different partitions of the divided input image are allocated to the CPU and GPU respectively to start parallel inference of the neural network, and then step 207 is executed.

[0039] In step 207, based on the operator settings and data partition size of the neural network layer, it is determined whether padding is required. If not, no padding operation is needed, and the process proceeds directly to step 209; if so, step 208 is executed.

[0040] In step 208, based on the operator settings of the neural network layer and the data partition size, the CPU and GPU partitions are padded with zeros of a certain size, and then step 209 is executed.

[0041] In step 209, CPU-GPU parallel inference is performed on this layer of neural network, and then step 210 is executed.

[0042] In step 210, it is determined whether the next layer of the neural network is an output layer. If not, the process proceeds to step 207 to perform parallel inference of the neural network. If it is, step 211 is executed.

[0043] In step 211, the output feature maps of the CPU and GPU are merged into the CPU or GPU, thus completing the CPU-GPU parallel inference, and the process ends in step 212.

[0044] This invention ensures that the neural network only needs to be computed locally on the CPU and GPU by locally padding necessary pseudo-data on the CPU and GPU. This significantly reduces the data sharing overhead of CPU-GPU parallel inference and substantially lowers inference latency. Furthermore, by utilizing a boundary-aware partitioning method, a latency-accuracy balanced CPU-GPU parallel inference is achieved.

[0045] The above is merely a further description of the present invention and is not intended to limit the scope of this patent. Any equivalent implementation of the present invention should be included within the scope of the claims of this patent.

Claims

1. A CPU-GPU parallel inference method for intelligent terminals, characterized in that, A CPU-GPU parallel inference approach that balances latency and accuracy is achieved by employing local padding and boundary awareness methods. The specific methods include: Step 1: Input an image that requires neural network inference; Step 2: Based on the actual latency of neural network inference on CPU and GPU, obtain the optimal image segmentation ratio, and combine it with the segmentation dimension to form the candidate segmentation pattern of the input image; Step 3: Based on the edge features extracted by edge detection, determine whether it is the partitioning mode with the fewest edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, it is the optimal partitioning mode and the following steps are executed. Step 4: Divide the image according to the optimal partitioning mode selected above, and allocate the different partitions to the CPU and GPU respectively for parallel inference of the neural network; Step 5: Based on the operator settings and data partition size of this neural network layer, determine whether pseudo-data needs to be filled. If no filling is needed, proceed directly to Step 7; if filling is needed, execute the following steps. Step 6: Based on the operator settings and data partition size of this neural network layer, pad the CPU and GPU partitions with zeros of a certain amount, and then execute the following steps; Step 7: Perform CPU-GPU parallel inference for this layer of the neural network; Step 8: Determine if the next layer of the neural network is an output layer. If not, return to step 5 and perform parallel inference of the neural network again. If it is, execute the following steps. Step 9: Merge the output feature maps of the CPU and GPU, and input them into the CPU or GPU to complete CPU-GPU parallel inference with a balance between latency and accuracy.

2. The intelligent terminal CPU-GPU parallel inference method according to claim 1, characterized in that, The pseudo-data filling in step five is determined by the operator settings and data partition size of each layer of the neural network. The pseudo-data is then filled locally by the processor, so that each layer of the neural network can be executed in parallel locally on the CPU and GPU.

3. The intelligent terminal CPU-GPU parallel inference method according to claim 1 or claim 2, characterized in that, The pseudo-data is padded with zeros.

Images