High-throughput JPEG (Joint Photographic Experts Group) heterogeneous reasoning and hybrid parallel method on mobile equipment

Abstract

The invention discloses a high-throughput JPEG (Joint Photographic Experts Group) heterogeneous reasoning and hybrid parallel method on mobile equipment, and provides an FCG reasoning framework aiming at the problems of high decoding time consumption proportion, high cross-processor communication overhead and low processor resource utilization rate in the existing mobile terminal JPEG image recognition. And optimization is carried out by combining JPEG frequency domain information with characteristics of a heterogeneous processor of a mobile terminal. The framework comprises an off-line preprocessing stage, an off-line task division stage and an on-line task scheduling stage, wherein in the off-line preprocessing stage, processor operation data are collected, and a delay prediction model is established; in the off-line task division stage, a processor cluster is constructed, model tasks are distributed, and the task amount is adjusted; and in the online task scheduling stage, hybrid parallel pipeline identification is executed according to the scheme. According to the method, through multi-core Huffman decoding, heterogeneous network structure design and optimization of task scheduling, the identification delay is remarkably reduced, the throughput is improved, the energy consumption is reduced, and the batch JPEG image identification performance of the mobile terminal is effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning inference and mobile image processing, and particularly relates to a high-throughput JPEG heterogeneous inference and hybrid parallel method on a mobile device. BACKGROUND

[0002] Deep neural networks (DNN) have been widely used in mobile object recognition and analysis due to their powerful feature extraction capabilities, enabling high-precision object classification and detection. Due to storage space and bandwidth limitations, images on mobile devices are usually stored and transmitted in JPEG format. Mobile JPEG image recognition scenarios include local image classification on mobile phones and data processing for mobile cameras.

[0003] Existing mainstream deep learning inference frameworks for mobile devices (such as Tensorflow, NCNN, and MACE) use sequential processing for batch JPEG image recognition. The recognition process for each image includes JPEG decoding (conversion to RGB data) and DNN model inference. However, this approach has three major challenges: 1) High time consumption for JPEG decoding: The time consumption of the JPEG decoding process usually exceeds the model inference time, accounting for more than 50% of the entire recognition process. Existing optimizations focus on the inference stage and do not comprehensively optimize decoding and inference; 2) Large communication overhead across processors: To ensure the accuracy of operator operations, frequent data communication and merging are required when using multiple processors for acceleration. Excessive communication overhead can offset the speed improvement brought by multiple processors. Methods such as CODL and NN-stretch utilize multiple processor resources but fail to effectively address the communication issue; 3) Low utilization of processor resources: In the sequential processing mode, JPEG decoding usually only uses a single processor core, while model inference can fully utilize resources, resulting in idle resources during the decoding phase and significant waste of resources on mobile devices.

[0004] In addition, some existing technologies such as Bigarm only use CPU large and small cores for pipeline inference and do not involve GPU resources. Model quantization only accelerates inference by reducing parameters and does not consider the decoding process. Frequency domain learning methods such as NIPS use frequency domain data for recognition but do not adapt to mobile processor characteristics, still having decoding time consumption and resource idling issues. Therefore, there is an urgent need for a technical solution that can comprehensively optimize the decoding and inference process and adapt to heterogeneous processors to improve the throughput and efficiency of mobile JPEG image recognition. SUMMARY

[0005] In view of the defects of high decoding time consumption, large communication overhead and low resource utilization in the prior art of mobile terminal batch JPEG image recognition, the application provides a high-throughput JPEG heterogeneous inference and hybrid parallel method on a mobile device, which realizes the cooperative optimization of decoding and inference process, improves the processor resource utilization and recognition throughput, reduces the delay and energy consumption, and has good popularization and application prospect.

[0006] To achieve the above object, the application adopts the following technical solutions: A high-throughput JPEG heterogeneous inference and hybrid parallel method on a mobile device, comprising the following steps: S1, offline preprocessing stage: decoding and processing the images of the JPEG image dataset on the mobile terminal, executing and processing the operators of the OP dataset, collecting the Huffman decoding running time and neural network operator running time of different processor cores; based on the collected data, modeling the Huffman decoding time and operator running time respectively, and constructing a delay prediction model; S2, offline task division stage: all processors in the mobile device are allocated as several processor clusters, the deep learning model is allocated to the corresponding task pipeline stage according to the processor cluster characteristics and device heterogeneous architecture characteristics; based on the delay prediction model, the performance of each processor and the model accuracy constraint are combined to adjust the task quantity of the double-branch part of the model, so that each processor cluster and different processors in the cluster complete the task synchronously; S3, online task scheduling stage: according to the DCT model, the task quantity of each pipeline stage and the processor allocation scheme determined in the task division stage, the batch JPEG image recognition task is allocated to the corresponding processor, and the hybrid parallel pipeline recognition is performed.

[0007] Preferably, in the offline preprocessing stage of step S1, the Huffman decoding adopts a multi-core parallel processing mode, and the decoding time consumption is reduced through scanning and multi-core cooperative decoding.

[0008] Preferably, in the offline task division stage of step S2, the division of the processor cluster is based on the calculation performance, architecture difference and task processing adaptability of each processor in the mobile device.

[0009] Preferably, in the offline task division stage of step S2, three design principles are followed: minimizing the waiting time of different task pipeline stages to reduce pipeline bubbles; minimizing the waiting time between different processors in the same task pipeline stage to reduce heterogeneous pipeline branch bubbles; and improving the running efficiency of each task pipeline stage to shorten the pipeline execution time.

[0010] Preferably, in the offline task partitioning phase of step S2, the modeling process of the delay prediction model includes data cleaning, feature extraction and regression analysis to achieve accurate prediction of decoding and inference time of different processor cores.

[0011] Preferably, in the online task scheduling phase of step S3, the DCT model is a heterogeneous network model constructed based on DCT frequency domain data, without the need to convert JPEG image decoding into RGB data, and directly uses JPEG frequency domain information for identification.

[0012] After adopting the above technical solution, the present application has the following beneficial effects: 1. The present application proposes an FCG inference framework, adopts a hybrid parallel pipeline method, and optimizes the JPEG frequency domain information and the mobile terminal heterogeneous processor characteristics. The framework includes three stages of offline preprocessing, offline task partitioning and online task scheduling: the offline preprocessing stage collects processor running data and establishes a delay prediction model; the offline task partitioning stage constructs a processor cluster, allocates model tasks and adjusts the task quantity; and the online task scheduling stage executes the hybrid parallel pipeline identification according to the scheme. Through multi-core Huffman decoding, heterogeneous network structure design and optimized task scheduling, the present application significantly reduces the identification delay, improves the throughput and reduces the energy consumption, and effectively improves the performance of mobile terminal batch JPEG image identification.

[0013] 2. The present application reduces the decoding time consumption through multi-core Huffman decoding, optimizes the communication overhead in combination with heterogeneous inference, and the FCG framework reduces the delay by 36% compared with the NN-Stretch, and the delay is significantly reduced. In the CPU, GPU and CPU+GPU environment, the image decoding throughput is 3.6 times, 3.3 times and 2.8 times of the traditional method respectively, and the throughput is greatly improved. When 100 JPEG images are continuously inferred, compared with the RGB sequence identification on the CPU, GPU and CPU+GPU, the energy consumption is reduced by 56%, 35% and 43% respectively, and 18% energy is saved compared with the DCT sequence identification. Through the hybrid parallel pipeline and task scheduling, the processor resources in the decoding and inference stages are fully utilized, idle waste is avoided, the characteristics of limited mobile terminal resources are adapted, and the resource utilization rate is improved. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 The flowchart of the present application. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application is further described in detail below in combination with embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0016] As Figure 1As shown, a high-throughput JPEG heterogeneous inference and hybrid parallel method on a mobile device includes the following steps: S1, offline preprocessing stage: decoding the images of the JPEG image dataset on the mobile terminal, executing the operators of the OP dataset, collecting the Huffman decoding running time and neural network operator running time of different processor cores; based on the collected data, the Huffman decoding time and operator running time are modeled respectively, and a delay prediction model is constructed; In the offline preprocessing stage of step S1, Huffman decoding adopts multi-core parallel processing mode, which reduces decoding time consumption through scanning and multi-core cooperative decoding; S2, offline task division stage: all processors in the mobile device are allocated as several processor clusters, and the deep learning model is allocated to the corresponding task pipeline stage according to the processor cluster characteristics and device heterogeneous architecture characteristics; based on the delay prediction model, the performance of each processor and the model accuracy constraint are combined to adjust the task quantity of the double-branch part of the model, so that each processor cluster and different processors in the cluster complete the task synchronously; In the offline task division stage of step S2, the division of the processor cluster is based on the calculation performance, architecture difference and task processing adaptability of each processor in the mobile device; In the offline task division stage of step S2, three design principles are followed: minimizing the waiting time of different task pipeline stages to reduce pipeline bubbles; minimizing the waiting time between different processors in the same task pipeline stage to reduce heterogeneous pipeline branch bubbles; and improving the running efficiency of each task pipeline stage to shorten the pipeline execution time; In the offline task division stage of step S2, the modeling process of the delay prediction model includes data cleaning, feature extraction and regression analysis to accurately predict the decoding and inference time of different processor cores; S3, online task scheduling stage: according to the DCT model, the task quantity of each pipeline stage and the processor allocation scheme determined in the task division stage, the batch JPEG image recognition task is allocated to the corresponding processor, and the hybrid parallel pipeline recognition is executed; In the online task scheduling stage of step S3, the DCT model is a heterogeneous network model constructed based on DCT frequency domain data, which does not need to convert JPEG image decoding into RGB data, but directly uses JPEG frequency domain information for recognition. Performance tests:

[0017] 1, offline preprocessing stage Data collection: Select 1000 JPEG images from the ImageNet dataset as the test set to perform decoding operations; select a common set of DNN operators (such as convolution, pooling, and activation) as the OP dataset, and execute them on the device's CPU large core, small core, and GPU, respectively. Use performance monitoring tools to record the Huffman decoding time and operator running time of each processor core, and collect 1000 sets of valid data; Model construction: Perform outlier rejection and normalization on the collected data, extract processor type, task data volume, and other features, and use a random forest regression algorithm to construct a delay prediction model. The prediction error of the model for decoding time is less than 5%, and the prediction error for operator running time is less than 8%.

[0018] 2. Offline task division stage Processor cluster division: Divide the 4 CPU large cores into cluster 1, the 4 CPU small cores into cluster 2, and the GPU into cluster 3; cluster 1 is responsible for multi-core Huffman decoding, cluster 2 is responsible for DCT data preprocessing, and cluster 3 is responsible for model backbone reasoning; Model allocation: Split the MobileNet model based on DCT into a preprocessing branch and a backbone reasoning branch, assign the preprocessing branch to cluster 2, assign the backbone reasoning branch to cluster 3, and cluster 1 is only responsible for Huffman decoding tasks; Task volume adjustment: Use the delay prediction model to predict, if the preprocessing task of cluster 2 takes longer than the decoding task of cluster 1 by 0.2ms, adjust the small core task allocation of cluster 2, add 1 small core to participate in preprocessing, and make the two-stage time consumption synchronous to 1.5ms; similarly, adjust the task rhythm of cluster 3 and other clusters.

[0019] 3. Online task scheduling stage Task distribution: input 100 batch JPEG images into the system in a pipeline manner, and cluster 1's 4 CPU large cores simultaneously perform multi-core Huffman decoding on different images; Parallel processing: after cluster 1 completes decoding, it transfers DCT data to the CPU small core of cluster 2 for preprocessing, while cluster 1 starts decoding the next batch of images; after cluster 2 completes preprocessing, it transfers data to the GPU of cluster 3 for model reasoning, realizing pipeline parallelism of "decoding-preprocessing-reasoning"; Data communication: in the heterogeneous reasoning stage, cluster 2 and cluster 3 only share DCT data once, avoiding frequent communication overhead, and finally output the image classification result.

[0020] Implementation effect: in the embodiment, the total delay of the recognition of 100 batches of JPEG images is 5.32s, which is reduced by 36% compared with 8.31s of the traditional Tensorflow framework; the throughput reaches 18.8FPS, which is 4.8 times of the traditional scheme (3.9FPS); the whole energy consumption is 280mWh, which is reduced by 43% compared with the CPU+GPU RGB recognition scheme (491mWh), and the CPU and GPU resource utilization rates are kept above 90%, realizing the comprehensive optimization of delay, throughput and energy consumption.

[0021] The above merely describes a preferred specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A high-throughput JPEG heterogeneous inference and hybrid parallel method for mobile devices, characterized in that, Includes the following steps: S1. Offline preprocessing stage: Decode the images in the JPEG image dataset on the mobile device, execute the operators in the OP dataset, and collect the Huffman decoding runtime and neural network operator runtime of different processor cores. Based on the collected data, the Huffman decoding time and operator running time are modeled to construct a delay prediction model; S2, Offline Task Partitioning Stage: All processors in the mobile device are divided into several processor clusters. Based on the characteristics of the processor clusters and the heterogeneous architecture of the device, the deep learning model is assigned to the corresponding task pipeline stage. Based on the latency prediction model, combined with the performance of each processor and the model accuracy constraints, the workload of the dual-branch part of the model is adjusted so that each processor cluster and different processors within the cluster can complete the task synchronously. S3. Online Task Scheduling Phase: Based on the DCT model determined in the task partitioning phase, the workload of each pipeline phase, and the processor allocation scheme, the batch JPEG image recognition tasks are allocated to the corresponding processors to perform hybrid parallel pipeline recognition.

2. The high-throughput JPEG heterogeneous inference and hybrid parallel method on mobile devices as described in claim 1, characterized in that: In the offline preprocessing stage of step S1, Huffman decoding adopts a multi-core parallel processing method, which reduces decoding time by scanning and multi-core collaborative decoding.

3. The high-throughput JPEG heterogeneous inference and hybrid parallel method on mobile devices as described in claim 1, characterized in that: In the offline task partitioning stage of step S2, the processor clusters are partitioned based on the computing performance, architectural differences, and task processing adaptability of each processor in the mobile device.

4. The high-throughput JPEG heterogeneous inference and hybrid parallel method on mobile devices as described in claim 1, characterized in that, In the offline task partitioning stage of step S2, three design principles are followed: minimize the waiting time of different task pipeline stages to reduce pipeline bubbles; minimize the waiting time between different processors within the same task pipeline stage to reduce heterogeneous pipeline branch bubbles. Improve the efficiency of each task pipeline stage to shorten pipeline execution time.

5. The high-throughput JPEG heterogeneous inference and hybrid parallel method on mobile devices as described in claim 1, characterized in that: In the offline task partitioning stage of step S2, the modeling process of the latency prediction model includes data cleaning, feature extraction and regression analysis to achieve accurate prediction of the decoding and inference time of different processor cores.

6. The high-throughput JPEG heterogeneous inference and hybrid parallel method on mobile devices as described in claim 1, characterized in that: In the online task scheduling stage of step S3, the DCT model is a heterogeneous network model built based on DCT frequency domain data. It does not require decoding the JPEG image into RGB data, but directly uses the JPEG frequency domain information for recognition.

Images