CPU-GPU parallel reasoning method for intelligent terminal

Abstract

The invention discloses an intelligent terminal CPU-GPU (central processing unit-graphics processing unit) parallel reasoning method, which is characterized in that a local filling and division boundary sensing method is adopted to realize delay and precision balanced CPU-GPU parallel reasoning, and the method specifically comprises the following steps of: determining an optimal division mode for an input picture according to edge features of division boundaries; each image partition is delivered to a CPU and a GPU for parallel reasoning of the neural network, finally, merging is carried out in front of an output layer of the neural network, and necessary pseudo data is locally filled in front of some layers of the neural network. Compared with the prior art, the method has the advantages that by locally filling pseudo data, the data sharing overhead of CPU-GPU parallel reasoning is greatly reduced, and the reasoning delay is remarkably reduced. Besides, by utilizing a division boundary sensing division method, delay-precision balanced CPU-GPU parallel reasoning is realized, the method is simple and convenient, the use effect is good, and the method is not only suitable for an intelligent terminal, but also suitable for a personal computer and a server, and has a good application prospect.

Description

Technical Field

[0001] The present invention relates to the field of neural network reasoning technology, and in particular to a smart terminal CPU-GPU parallel reasoning method based on local filling and image partition boundary perception. Background Art

[0002] With the surge in deep learning-based applications in smart terminals, such as photo beautification, face detection, and object recognition, major Internet companies have launched deep learning frameworks for smart terminals, such as TensorflowLite, MACE, and NCNN. Running the entire neural network on a local device can effectively avoid the risk of user privacy leakage and network signal fluctuations, so on-device reasoning is becoming increasingly popular. Generally speaking, the current mainstream deep learning frameworks usually use one processor, such as a GPU, to execute the entire neural network, which loses the opportunity to accelerate using existing heterogeneous processor resources. In addition, due to size and power consumption, the performance of mobile GPUs is not as good as that of traditional servers. Mobile users always accuse CNN-related services of not being able to respond quickly. In recent years, some studies have used existing heterogeneous processor resources in smart terminals to accelerate neural network reasoning. Some studies reduce latency by assigning different layers of the neural network to different processors, but each layer can only be calculated on the selected processor, and the processing of the neural network is essentially still serial.

[0003] The performance gap of neural network reasoning between the CPU and GPU of smart terminals in the prior art is not obvious, and the CPU plays a more important role in the neural network reasoning of smart terminals. Therefore, in order to further improve performance, some researchers have accelerated each layer of the neural network by coordinating the CPU and GPU in parallel. By dividing along the output channel or height dimension, different partitions are assigned to the CPU and GPU for parallel processing. However, the output of the previous layer of the neural network needs to be shared between processors in order to reason about the next layer in parallel, because convolution and pooling operations need to process data across the boundaries of image partitions. Cross-processor data sharing significantly increases reasoning latency, which results in the performance improvement of CPU and GPU parallel reasoning on smart terminals still not meeting user expectations, and even making it difficult to accelerate lightweight neural networks. Summary of the invention

[0004] The purpose of the present invention is to provide a CPU-GPU parallel reasoning method for intelligent terminals in view of the deficiencies of the prior art. By filling necessary pseudo data locally in the CPU and GPU, it is ensured that the neural network only needs to be calculated locally in the CPU and GPU without data sharing overhead. The method determines the optimal partitioning mode of the input image according to the edge features of the partition boundary, and each image partition is respectively handed over to the CPU and GPU for parallel reasoning of the neural network, and finally merged before the output layer of the neural network. Local filling realizes the CPU-GPU parallel reasoning of intelligent terminals that eliminates data sharing overhead. It is convenient and simple, greatly reduces the data sharing overhead of CPU-GPU parallel reasoning, and significantly reduces the reasoning delay. The partitioning method based on partition boundary perception realizes CPU-GPU parallel reasoning with delay-precision balance. The method is simple and has good use effect. It is not only suitable for intelligent terminals, but also for personal computers and servers, and has good application prospects and certain commercial development value.

[0005] The specific technical solution to achieve the purpose of the present invention is: a CPU-GPU parallel reasoning method for an intelligent terminal, which adopts a local filling and partition boundary perception method to achieve CPU-GPU parallel reasoning with latency-precision balance, and the method specifically includes:

[0006] Step 1: Input a picture that needs neural network inference;

[0007] Step 2: According to the actual latency of the neural network in CPU and GPU reasoning, obtain the optimal image partition ratio (which can be the reciprocal of each other, such as 2:3 and 3:2), and combine it with the partition dimension to form a candidate partition mode for the input image;

[0008] Step 3: Based on the edge features extracted by edge detection, determine whether it is a partitioning mode with the least edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, select the optimal partitioning mode and execute the following steps;

[0009] Step 4: Divide the image according to the optimal partitioning mode selected above, and assign different partitions to the CPU and GPU for parallel reasoning of the neural network;

[0010] Step 5: According to the operator settings of the neural network layer and the size of the data partition, determine whether it is necessary to fill in dummy data. If not, go directly to step 7; if filling is required, execute the following steps;

[0011] Step 6: According to the operator settings of the neural network layer and the data partition size, fill the CPU and GPU partitions with a certain fill size of zeros, and then execute the following steps;

[0012] Step 7: Execute CPU-GPU parallel reasoning of the neural network layer;

[0013] Step 8: Determine whether the next layer of the neural network is the output layer. If not, return to step 5 to perform parallel reasoning of the neural network again; if yes, execute the following steps;

[0014] Step 9: Merge the output feature maps of the CPU and GPU and input them into the CPU or GPU to complete CPU-GPU parallel reasoning with balanced latency and accuracy.

[0015] The CPU and GPU processors fill in the necessary dummy data locally according to the operator settings of each layer of the neural network and the size of the data partition, accurately determine the size of the filling, and fill in the necessary dummy data locally in the CPU and GPU processors, so that each layer of the neural network is all calculated in parallel locally in the CPU and GPU, without the need for data sharing between the CPU and GPU. In addition, considering that there are hundreds or even thousands of feature maps in the neural network reasoning process, due to the hierarchical nature of feature extraction in the convolutional layer, no existing filling method is suitable for all feature maps. Inappropriate filling methods may result in high overhead or low accuracy. Therefore, 0 filling, which is widely used in neural networks, is adopted, which hardly introduces any additional overhead.

[0016] The local filling of necessary pseudo data only loses part of the information across the partition boundary. Considering that there may be a large number of redundant pixels in the image that are irrelevant to accurate recognition, we can use intelligent partitioning to make the lost information as little as possible to affect the accurate recognition of the image. In order to avoid the loss of information in important feature areas, the present invention selects the optimal partitioning mode with the least edge pixels in a certain interval near the candidate partition boundary based on the edge features extracted by edge detection. After partitioning according to the optimal partitioning mode, the different partitions of the input image are handed over to the CPU and GPU for parallel reasoning of the neural network.

[0017] Compared with the prior art, the present invention has the advantages of greatly reducing the data sharing overhead of CPU-GPU parallel reasoning and significantly reducing the reasoning delay by filling pseudo data locally. In addition, the partitioning method based on partition boundary awareness is used to achieve CPU-GPU parallel reasoning with latency-precision balance. The method is simple and has good use effect. It is not only applicable to smart terminals, but also to personal computers and servers, and has good application prospects and certain commercial development value. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG1 is a flow chart of the present invention;

[0019] Figure 2 is a schematic diagram of the candidate partitioning mode. DETAILED DESCRIPTION

[0020] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0021] Example 1

[0022] The optimal partitioning mode is determined for the input image based on the edge features of the partition boundary. Each image partition is sent to the CPU and GPU for parallel reasoning of the neural network, and finally merged before the output layer of the neural network. The necessary pseudo data is locally filled before certain layers of the neural network to achieve CPU-GPU parallel reasoning with latency-precision balance. The specific steps are as follows:

[0023] Step 1: Input a picture that needs neural network inference;

[0024] Step 2: According to the actual latency of the neural network in CPU and GPU reasoning, the optimal image partition ratio is obtained (the ratio can be the reciprocal of each other, such as 2:3 and 3:2), and it is combined with the partition dimension to form a candidate partition mode for the input image;

[0025] Step 3: Based on the edge features extracted by edge detection, determine whether it is a partitioning mode with the least edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, select the optimal partitioning mode and execute the following steps;

[0026] Step 4: Divide the image according to the optimal partitioning mode selected above, and assign different partitions to the CPU and GPU for parallel reasoning of the neural network;

[0027] Step 5: According to the operator settings of the neural network layer and the size of the data partition, determine whether it is necessary to fill in dummy data. If not, go directly to step 7; if filling is required, execute the following steps;

[0028] Step 6: According to the operator settings of the neural network layer and the data partition size, fill the CPU and GPU partitions with a certain fill size of zeros, and then execute the following steps;

[0029] Step 7: Execute CPU-GPU parallel reasoning of the neural network layer;

[0030] Step 8: Determine whether the next layer of the neural network is the output layer. If not, return to step 5 to perform parallel reasoning of the neural network again; if yes, execute the following steps;

[0031] Step 9: Merge the output feature maps of the CPU and GPU and input them into the CPU or GPU to complete CPU-GPU parallel reasoning with balanced latency and accuracy.

[0032] The filling of dummy data in step five is determined according to the operator settings of each layer of the neural network and the data partition size, and the dummy data is filled locally by the processor, so that each layer of the neural network is executed in parallel locally on the CPU-GPU without the need for data sharing between the CPU-GPU.

[0033] See also Figure 1 , the process of achieving CPU-GPU parallel reasoning with balanced latency and accuracy starts at step 201 and then executes step 202 .

[0034] In step 202, a picture requiring neural network inference is input, and step 203 is executed.

[0035] See also Figure 2 In step 203, the optimal image partition ratio (which can be reciprocal, such as 2:3 and 3:2) is obtained according to the actual delay of the neural network in CPU and GPU reasoning, and the candidate partition mode of the input image is formed together with the partition dimension, and then step 204 is executed.

[0036] In step 204, based on the edge features extracted by edge detection, it is determined whether it is an optimal partitioning mode with the least edge pixels in a certain interval near the candidate partitioning boundary. If not, this partitioning mode is discarded and the process directly proceeds to step 212 to end. If yes, step 205 is executed.

[0037] In step 205, the picture is divided according to the selected optimal division mode, and then step 206 is executed.

[0038] In step 206, different partitions of the divided input image are respectively allocated to the CPU and GPU to start parallel reasoning of the neural network, and then step 207 is executed.

[0039] In step 207, it is determined whether padding is required according to the operator setting of the neural network layer and the size of the data partition. If not, no padding operation is required and the process directly proceeds to step 209; if required, step 208 is executed next.

[0040] In step 208, according to the operator setting of the neural network layer and the data partition size, the CPU and GPU partitions are filled with zeros of a certain filling size, and then step 209 is executed.

[0041] The step 209 performs CPU-GPU parallel reasoning of the neural network layer, and then executes step 210.

[0042] The step 210 determines whether the next layer of the neural network is the output layer. If not, the process goes to step 207 again to perform parallel reasoning of the neural network. If yes, the process goes to step 211.

[0043] The step 211 combines the output feature maps of the CPU and GPU into the CPU or GPU, thus completing the CPU-GPU parallel reasoning, and then proceeds to step 212 to end.

[0044] The present invention ensures that the neural network only needs to be calculated locally on the CPU and GPU by filling necessary pseudo data locally on the CPU and GPU, greatly reducing the data sharing overhead of CPU-GPU parallel reasoning and significantly reducing reasoning latency. The partitioning method based on partition boundary awareness is used to achieve CPU-GPU parallel reasoning with latency-accuracy balance.

[0045] The above is only a further explanation of the present invention and is not intended to limit this patent. Any equivalent implementation of the present invention should be included in the scope of the claims of this patent.

Claims

1. A CPU-GPU parallel reasoning method for an intelligent terminal, characterized in that: The local filling and partition boundary awareness methods are used to achieve CPU-GPU parallel reasoning with balanced latency and accuracy. The specific methods include: Step 1: Input a picture that needs neural network inference; Step 2: According to the actual latency of the neural network in CPU and GPU reasoning, the optimal image partition ratio is obtained, and it is combined with the partition dimension to form a candidate partition pattern for the input image; Step 3: Based on the edge features extracted by edge detection, determine whether it is a partitioning mode with the least edge pixels in a certain interval near the candidate partitioning boundary. If not, discard the candidate partitioning mode; if so, select the optimal partitioning mode and execute the following steps; Step 4: Divide the image according to the optimal partitioning mode selected above, and assign different partitions to the CPU and GPU for parallel reasoning of the neural network; Step 5: According to the operator settings of the neural network layer and the size of the data partition, determine whether it is necessary to fill in dummy data. If not, go directly to step 7; if filling is required, execute the following steps; Step 6: According to the operator settings of the neural network layer and the data partition size, fill the CPU and GPU partitions with a certain fill size of zeros, and then execute the following steps; Step 7: Execute CPU-GPU parallel reasoning of the neural network layer; Step 8: Determine whether the next layer of the neural network is the output layer. If not, return to step 5 to perform parallel reasoning of the neural network again; if yes, execute the following steps; Step 9: Merge the output feature maps of the CPU and GPU and input them into the CPU or GPU to complete CPU-GPU parallel reasoning with balanced latency and accuracy.

2. According to the intelligent terminal CPU-GPU parallel reasoning method of claim 1, it is characterized in that: The filling of dummy data in step five is determined according to the operator settings of each layer of the neural network and the data partition size, and the dummy data is filled locally by the processor, so that each layer of the neural network is executed in parallel locally on the CPU-GPU.

3. The intelligent terminal CPU-GPU parallel reasoning method according to claim 1 or claim 2, characterized in that: The dummy data is filled with zeros.

Images