A Heterogeneous Edge Multi-Task Inference Method and Device Based on Pipeline Optimization

Abstract

The present invention discloses a heterogeneous edge-end multi-task reasoning method and device based on pipeline optimization. The present invention is based on pipeline stage division, balances computing load, adopts contention mitigation optimization method to adjust the execution order of contention model, and dynamically adjusts the workload and tail cavitation between pipeline stages through load stealing technology, minimizes the idle time cavitation of heterogeneous processors, and improves processing efficiency. By designing a pipeline parallel mechanism on heterogeneous processors and combining optimization strategies such as model segmentation, dynamic programming, and local search, the present invention improves the utilization rate of heterogeneous computing resources, including CPU large cores, CPU small cores, OpenCL GPUs, etc., thereby achieving a significant increase in computing throughput and completing the efficient utilization of terminal-side computing resources.

Description

Technical Field

[0001] The present invention relates to the field of edge inference technology, and in particular to a heterogeneous edge multi-task inference method and device based on pipeline optimization. Background Art

[0002] Edge computing and cloud computing differ significantly in hardware architecture, computing power, programming difficulty, inference speed, and data security. Cloud computing, relying on its powerful centralized computing and storage capabilities, is well-suited for large-scale model training and complex tasks. However, due to network transmission limitations, it can face high latency and data security risks. In contrast, edge computing hardware is fragmented and has relatively limited computing resources. While better suited to local real-time processing needs, it presents both challenges and opportunities in terms of programming complexity and inference efficiency. In particular, the heterogeneity of edge hardware complicates system design, making efficient multi-task parallel processing and resource utilization difficult. Summary of the Invention

[0003] The purpose of this invention is to improve and standardize the deficiencies in existing research and technology, and to propose a heterogeneous edge multi-task reasoning method and device based on pipeline optimization.

[0004] The objectives of the present invention are achieved through the following technical solutions: In a first aspect, the present invention provides a heterogeneous edge multi-task reasoning method based on pipeline optimization, comprising the following steps:

[0005] Step 1: Input preprocessing and DNN request definition: Obtain input data and preprocess it, and select several corresponding deep neural network models based on task requirements;

[0006] Step 2: Execution on heterogeneous computing platforms: Multiple deep neural network models are assigned to corresponding heterogeneous processors for inference execution, and represented by pipeline stage division;

[0007] Step 3, horizontal model splitting: Using the pipeline stage division in step 2, for each request in the deep neural network model, the goal is to find a partitioning strategy that minimizes inference latency and balances the computational load across different pipeline stages.

[0008] Step 4: Contention mitigation optimization: Based on the partitioning strategy for each deep neural network model obtained in step 3, a resource contention window is established. Linear regression of performance detection parameters is used to classify high-contention and low-contention models. The execution order of high-contention models is adjusted to avoid repeated high-contention models within the window.

[0009] Step 5: Minimize pipeline bubbles: Using the final model execution order obtained in Step 4, dynamically adjust the workload and tail bubbles between pipeline stages through load stealing technology to minimize the idle time bubbles of heterogeneous processors and improve processing efficiency.

[0010] Furthermore, in step 1, the input data includes images, video slices and voice data.

[0011] Furthermore, in step 2, the set of layers of the i-th deep neural network model is segmented into subsets, each subset represents a specific part of the divided layer, and these parts are processed in various stages of the pipeline to achieve optimized load distribution between heterogeneous processors.

[0012] Furthermore, in step 3, the problem of minimizing the inference delay is converted into a MinMax problem based on the inference delay function of the sub-model of model i on processor k, and a dynamic programming algorithm is used to slice the deep neural network model to obtain the optimal substructure.

[0013] Furthermore, in step 4, the specific method for optimizing contention mitigation is as follows: first, a method for quantifying contention intensity is defined, valid performance events are used as a feature set, and a regularization term is introduced to construct a regression model to mitigate overfitting, thereby predicting contention intensity; model requests are temporally staggered based on contention intensity;

[0014] To implement pipeline scheduling of models on K heterogeneous processors, a contention window needs to be defined for each model. Based on a preset contention intensity threshold, the window is divided into high-contention models and low-contention models. The contention distance between multiple high-contention slice indexes in the window will lead to time overlap, and low-contention slices need to be relocated from the low-contention model to eliminate contention. This is converted into a linear allocation problem (LAP) and optimized with the goal of minimizing the total movement cost.

[0015] Furthermore, in step 5, for each column of parallel processes in the pipeline parallel stage, the size of the pipeline bubble and the inference latency are calculated, and the total idle time bubble is minimized through a dynamic optimization method of work stealing. The optimization process includes the following two stages:

[0016] 1) Perform work stealing within each contention window and slide the window until the end of the sequence;

[0017] 2) Minimize tail bubbles through local search; in stage 1), for the model within the contention window, first find the critical path; then perform work stealing in layers.

[0018] Furthermore, in the dynamic optimization phase of work stealing, the execution time between different stages is vertically aligned to minimize pipeline bubbles, and batch size is used for pipeline alignment optimization.

[0019] In the second aspect, the present invention also provides a heterogeneous edge multi-task reasoning device based on pipeline optimization, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it implements the heterogeneous edge multi-task reasoning method based on pipeline optimization.

[0020] In a third aspect, the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the heterogeneous edge multi-task reasoning method based on pipeline optimization is implemented.

[0021] In a fourth aspect, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the heterogeneous edge multi-task reasoning method based on pipeline optimization.

[0022] Compared with the prior art, the present invention has the following innovative advantages and significant effects:

[0023] 1) This invention innovatively considers co-execution deceleration in resource-constrained edge device pipeline planning, meeting the current complex and high-speed operating environment at the edge and the development needs of cloud-edge-end collaboration;

[0024] 2) Incorporating a contention window, it reorders incoming requests with minimal displacement cost. This transforms the complex contention mitigation problem into a classic linear allocation problem, employing a polynomial-time solution. Work stealing is performed between different pipeline stages to minimize pipeline bubbles and improve resource utilization. Extensive evaluation on commercial SoCs demonstrates excellent speedup.

[0025] 3) Based on in-depth research in related fields, this paper addresses the challenges of edge computing by breaking through traditional mobile CPU-centric system designs and enabling efficient parallel scheduling of various computing units (such as CPUs, GPUs, and NPUs) on terminal devices, effectively alleviating the problem of CPU resource utilization. By designing a pipeline parallel mechanism on heterogeneous processors and combining optimization strategies such as model partitioning, dynamic programming, and local search, it improves the utilization of heterogeneous computing resources, including large and small CPU cores, OpenCL GPUs, and others. This significantly increases computing throughput and effectively utilizes terminal-side computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1This is a flow chart of the pipeline-optimized heterogeneous edge multi-task inference method of the present invention.

[0027] Figure 2 It is a schematic diagram of the pipeline cavitation lateral optimization, load stealing and tail optimization of the present invention.

[0028] Figure 3 This is a comparison of the final inference latency of the method of the present invention (heterogeneous pipeline) and two benchmark methods.

[0029] Figure 4 This is a structural diagram of a heterogeneous edge multi-task inference device based on pipeline optimization provided by the present invention. DETAILED DESCRIPTION

[0030] The specific implementation method and working principle of the present invention are described in detail below with reference to the accompanying drawings:

[0031] In existing consumer-grade mobile devices, the CPU-centric architecture design often leads to low utilization of computing resources. Traditional model-level parallel reasoning strategies usually perform parallel processing on different processors at the model-level granularity, without further subdividing the granularity to the network layer, and do not fully consider the bubbles that may appear during the reasoning process. Therefore, these two strategies often lead to waste of computing resources and queuing of reasoning tasks, which in turn generates large bubbles. In order to more effectively utilize processor resources and reduce bubbles, the embodiment of the present invention selects a single CPU large-core serial reasoning and a model-level parallel reasoning strategy as two benchmarks for comparison with the pipeline parallel architecture designed by the present invention.

[0032] In the context of the embodiment of the present invention, the influence of NPU is not considered for the time being. Instead, the CPU large core, OpenCL of mobile GPU, and CPU small core are used as the test order of the processor. The embodiment of the present invention provides a heterogeneous edge multi-task inference method based on pipeline optimization, covering 7 typical network models, including traditional CNN models (such as Vgg16, Yolov4, Squeezenet, Resnet50, Alexnet) and Transformer models (Bertsquad, Vitpose). The numbering order of each model is Vitpose, Squeezenet, Yolov4, Bertsquad, Alexnet, Resnet50, Vgg16, respectively, from 1 to 7. The test is carried out using the Kirin 990 chip platform as an example. The detailed implementation steps are as follows:

[0033] In step 1, the seven network models are preprocessed and DNN requests are defined. Preprocessing operations such as standardization, resizing, and normalization are performed on input data such as images, video slices, and speech. The corresponding DNN model inference requests are defined according to the downstream task requirements, and the appropriate model is selected for inference.

[0034] Step 2: The heterogeneous computing platform performs the computation, giving full play to the CPU multi-threading, GPU parallel computing and NPU efficient reasoning capabilities, and collaboratively performs DNN model reasoning to achieve optimal resource utilization. For each model, the set of layers of the i-th deep neural network model is represented as ; These are divided into the following subsets after model segmentation:

[0035]

[0036] Each subset represents a specific part of the partitioned layer, which is processed in various stages of the pipeline to achieve optimal load distribution among heterogeneous processors. DNN models are assigned to heterogeneous processors, and use Road pipeline stage division express.

[0037] Step 3: Use the basic division in step 2 For each request in the seven network models, a dynamic programming algorithm is used to slice and balance the computing load of each processor, minimizing the inference latency and improving computing efficiency. The problem of minimizing inference latency can be formulated as the following MinMax problem:

[0038]

[0039] in, Represented as a model In the processor The inference delay function of the above sub-model, Representation Model The sub-model of the deep neural network model is sliced using a dynamic programming algorithm to obtain the optimal sub-structure; define For the From layer 0 to layer 1 on a heterogeneous processor The minimum of the maximum execution time in the optimal partitioning of the layers, where and , K is the number of heterogeneous processors; Indicates the layer To layer In the Total inference time on each processor; It has the following optimal substructure:

[0040]

[0041] Its boundary conditions are:

[0042]

[0043] Algorithm initialization dynamic programming table ,use Monotonicity, according to the condition renew The value of , through recursive processing, determines the optimal execution time Once confirmed , then use the direct partitioning strategy backtracking through the dynamic programming table to determine the optimal set of partitioning points.

[0044] Taking the model arrangement order of 1234567 as an example, the exhaustive algorithm and the invented algorithm are compared. The specific segmentation is shown in Table 1:

[0045] Table 1 Comparison of 1234567 network combination segmentation (Intel i7)

[0046]

[0047] Step 4: Optimize contention mitigation. First, define a method to quantify contention intensity and stagger high-load model requests in time. To this end, use effective performance events as feature sets. , by introducing the α regularization term to build a regression model to alleviate overfitting, thus predicting the contention intensity ; Its optimization goal is defined as:

[0048]

[0049] Among them, the weight matrix The calculation formula is:

[0050]

[0051] Where, Represents the identity matrix. Based on this method, for a new inference request, its corresponding contention intensity is quickly estimated based on performance event statistics.

[0052] To realize the model Pipeline scheduling on heterogeneous processors requires each model Define the contention window (ranging from to ), based on the preset contention intensity threshold, the window is divided into high contention model and low contention model, and the and Represents the number of high contention and low contention model slices in the window, respectively. If , then the high contention slice index and Contention distance between This will cause time overlap, so you need to Reposition at least low contention slices to eliminate contention; transform the above problem into a linear assignment problem LAP and define the decision variables , Indicates that the The low contention slice is moved to high contention positions, otherwise 0, and minimize the total movement cost The present invention is based on the partitioning strategy obtained in step 3. First, based on contention intensity, we categorized Yolov4, Bertsquad, Vitpose, and Vgg16 as high-contention models, and the others as low-contention models. We then established a resource contention window and adjusted the execution order of high-contention models to avoid duplicate high-contention models within the window, reducing resource contention on the memory bus and lowering latency. We also performed sequential resource contention optimization on the seven network models, minimizing the possibility of two high-contention models appearing in a single contention window.

[0053]

[0054] in, and are the total number of low and high contention slices, respectively. The constraint ensures that each slice is allocated to a unique location; when exist when in the left or right contention window of Assigned to This will lead to high contention; it is solved by the Kuhn–Munkres algorithm:

[0055]

[0056] Step 5, perform the final pipeline bubble minimization, using the final model execution sequence obtained in step 4, such as Figure 2 As shown in the figure, the workload and tail bubbles between pipeline stages are dynamically adjusted by load stealing technology, minimizing the idle time bubbles of the seven-network pipeline parallel stage processors and improving processing efficiency. Specifically: heterogeneous processors Upper model The sub-model is represented as and use express The inference delay of

[0057] At the same time, each column of parallel process in the pipeline parallel stage is defined as ,in ; each The size of the pipeline cavitation in and inference latency Determined by:

[0058]

[0059]

[0060] Furthermore, it is necessary to minimize the total idle time bubbles and alleviate them through the dynamic optimization method of work stealing:

[0061]

[0062] The optimization process consists of two stages:

[0063] 1) Perform work stealing within each contention window CW and slide the window until the end of the sequence;

[0064] 2) Minimize the tail bubble by local search; in stage 1), for the model , first find the critical path ; then perform work stealing on a per-layer basis.

[0065] if , then slice the model The layers are redistributed to , making The goal is to vertically align the execution time between different stages to minimize pipeline bubbles, and use batch size to optimize pipeline alignment. In this paper, Yolov4 is used as the critical path, and the specific batch size settings are: 3 for Vgg16, 32 for SqueezeNet and AlexNet, and 8 for ResNet50.

[0066] For the comparison benchmark, in the model-level parallel strategy, a strategy of matching processors with higher computing power to tasks with larger demands was adopted. It should be noted that due to the existence of Batchsize, the size of the task is not directly equivalent to the size of the network. In the first stage, the CPU large core is responsible for inferring Yolov4, OpenCL is responsible for inferring Resnet50, and the CPU small core processes Alexnet; in the second stage, the CPU large core is packaged for inferring Bertsquad and Vitpose, OpenCL is responsible for Vgg16, and the CPU small core is inferring Squeezenet. Finally, a heterogeneous edge multi-task inference method based on pipeline optimization was invented, which showed significant improvement compared with the two benchmarks, as shown in Table 2 and Figure 3 As shown:

[0067] Table 2. Comparison of network combination inference latency without NPU 7654321 (Kirin 990) (unit: ms)

[0068]

[0069] Corresponding to the aforementioned embodiment of a heterogeneous edge multi-task reasoning method based on pipeline optimization, the present invention also provides an embodiment of a heterogeneous edge multi-task reasoning device based on pipeline optimization.

[0070] See also Figure 4 , an embodiment of the present invention provides a heterogeneous edge multi-task reasoning device based on pipeline optimization, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a heterogeneous edge multi-task reasoning method based on pipeline optimization in the above embodiment.

[0071] The embodiment of a heterogeneous edge-end multi-task reasoning device based on pipeline optimization provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, if Figure 4 As shown in the figure, it is a hardware structure diagram of any device with data processing capability where a heterogeneous edge multi-task reasoning device based on pipeline optimization provided by the present invention is located. Figure 4In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.

[0072] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0073] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present invention. A person of ordinary skill in the art can understand and implement the present invention without inventive work.

[0074] An embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a heterogeneous edge multi-task reasoning method based on pipeline optimization in the above embodiment is implemented.

[0075] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0076] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the heterogeneous edge multi-task reasoning method based on pipeline optimization.

[0077] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A heterogeneous edge multi-task reasoning method based on pipeline optimization, characterized by: The following steps are involved: Step 1: Input preprocessing and DNN request definition: Obtain input data and preprocess it, and select several corresponding deep neural network models based on task requirements; Step 2: Execution on heterogeneous computing platforms: Multiple deep neural network models are assigned to corresponding heterogeneous processors for inference execution, and represented by pipeline stage division; Step 3, horizontal model splitting: Using the pipeline stage division in step 2, for each request in the deep neural network model, the goal is to find a partitioning strategy that minimizes inference latency and balances the computational load across different pipeline stages. Step 4: Contention Mitigation Optimization: Based on the partitioning strategy for each deep neural network model obtained in Step 3, a resource contention window is established. Linear regression of performance detection parameters is used to classify high-contention and low-contention models. The execution order of high-contention models is adjusted to avoid repeated high-contention models within the window. The specific contention mitigation optimization method is as follows: First, a method for quantifying contention intensity is defined. Effective performance events are used as a feature set. Regularization terms are introduced to construct a regression model to mitigate overfitting, thereby predicting contention intensity. Stagger model requests in time based on contention intensity; To implement pipeline scheduling of models on K heterogeneous processors, a contention window must be defined for each model. Based on a preset contention intensity threshold, the window is divided into high-contention models and low-contention models. The contention distance between multiple high-contention slice indexes within the window will cause time overlap, and low-contention slices must be relocated from the low-contention model to eliminate contention. Convert it into a linear assignment problem LAP and optimize it with the goal of minimizing the total movement cost; Step 5: Minimize pipeline bubbles: Using the final model execution order obtained in Step 4, dynamically adjust the workload and tail bubbles between pipeline stages through load stealing technology to minimize the idle time bubbles of heterogeneous processors and improve processing efficiency.

2. The heterogeneous edge multi-task reasoning method based on pipeline optimization according to claim 1 is characterized in that: In step 1, the input data includes images, video slices and voice data.

3. The heterogeneous edge multi-task reasoning method based on pipeline optimization according to claim 1 is characterized in that: In step 2, the set of layers of the i-th deep neural network model is segmented into subsets, each subset represents a specific part of the divided layer, and these parts are processed in various stages of the pipeline to achieve optimized load distribution between heterogeneous processors.

4. The heterogeneous edge multi-task reasoning method based on pipeline optimization according to claim 1 is characterized in that: In step 3, the problem of minimizing the inference delay is converted into a MinMax problem based on the inference delay function of the sub-model of model i on processor k, and a dynamic programming algorithm is used to slice the deep neural network model to obtain the optimal substructure.

5. The heterogeneous edge multi-task reasoning method based on pipeline optimization according to claim 1 is characterized in that: In step 5, for each column of parallel processes in the pipeline parallel stage, the size of the pipeline bubble and the inference delay are calculated, and the total idle time bubble is minimized by a dynamic optimization method of work stealing; The optimization process consists of the following two stages: 1) Perform work stealing within each contention window and slide the window until the end of the sequence; 2) Minimize tail bubbles through local search; in stage 1), for the model within the contention window, first find the critical path; then perform work stealing in layers.

6. The heterogeneous edge multi-task reasoning method based on pipeline optimization according to claim 5 is characterized in that: In the dynamic optimization phase of work stealing, the execution time is vertically aligned between different stages to minimize pipeline bubbles, and batch size is used for pipeline alignment optimization.

7. A heterogeneous edge multi-task inference device based on pipeline optimization, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that: When the processor executes the executable code, it implements a heterogeneous edge multi-task reasoning method based on pipeline optimization according to any one of claims 1 to 6.

8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, a heterogeneous edge multi-task reasoning method based on pipeline optimization according to any one of claims 1 to 6 is implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a heterogeneous edge multi-task reasoning method based on pipeline optimization as described in any one of claims 1 to 6.

Images