CPU-GPU collaborative execution optimization method and system for deep learning reasoning of mobile equipment

Abstract

The invention relates to the technical field of mobile terminal data processing, in particular to a CPU-GPU collaborative execution optimization method and system for deep learning reasoning of mobile equipment. The method comprises the following steps: S1, an offline stage: defining complete steps of operator collaborative execution, including data conversion, mapping, pre-synchronization, calculation, post-synchronization and de-mapping; establishing an operator delay model by adopting a lightweight regression predictor, and calculating the total prediction delay T cooperatively executed by an operator; and S2, an online stage: constructing mixed type friendly data sharing to allow the heterogeneous processor to use different data types during reasoning, and reducing the number of operators needing to share data. The problems that an existing collaborative execution framework is large in data sharing overhead, unreasonable in workload partition, inaccurate in delay prediction or too high in complexity are solved, efficient CPU-GPU collaborative execution is achieved, the reasoning performance is improved, energy consumption is reduced, and the workload is balanced.

Description

Technical Field

[0001] This invention relates to the field of data processing technology for mobile devices, and in particular to an optimized method and system for CPU-GPU co-execution of deep learning inference in mobile devices. Background Technology

[0002] With the widespread application of deep learning in tasks such as object detection, semantic segmentation, and image style transfer, the demand for deploying deep learning models on mobile devices is increasing due to their advantages in privacy protection, network resilience, and low cloud operation overhead.

[0003] Current deep learning inference frameworks on mobile devices primarily execute in two ways: single-processor execution and partially attempted heterogeneous processor collaborative execution. In single-processor execution, for example, the MACE framework can perform model inference on either the CPU or GPU, selecting the appropriate data type based on the processor type (e.g., image types for Adreno GPUs, and buffer types for Mali GPUs and CPUs). Regarding heterogeneous processor collaborative execution, μlayer and Optic are representative examples, attempting to enable CPUs and GPUs to work together on mobile devices, accelerating deep learning inference through methods such as data quantization and workload partitioning.

[0004] The specific design of mobile System-on-Chip (SoC) provides the basis for enabling CPUs and GPUs to work together on mobile devices for two reasons: 1) Comparable CPU and GPU performance: Unlike server GPUs, which run several orders of magnitude faster than CPUs, mobile CPUs and GPUs have similar performance in deep learning inference, so they can run in parallel; 2) Unified memory: Unlike server machines that typically have separate memory for CPUs and GPUs, mobile CPUs and GPUs use unified memory, which avoids copying data between different memory locations.

[0005] From a software functional module perspective, these frameworks include modules such as model loading, operator execution, and data management. In terms of hardware support, they rely on the SoC of mobile devices, including the CPU and GPU. The general workflow is as follows: the framework loads the deep learning model, allocates the operators in the model to the corresponding processors for execution, and the execution process involves data transfer and processing between processors.

[0006] The execution of existing deep learning inference frameworks on mobile devices still faces the following main problems: Data sharing issues: Existing cooperative execution systems use uniform data types (such as buffer types) to simplify data sharing, which prevents the full utilization of each processor's performance. For example, on Adreno GPUs, using buffer types for convolution operations compared to image types significantly increases latency, making CPU + GPU cooperative execution slower than using the GPU alone. Furthermore, the overhead of data sharing is not adequately considered, including the overhead of data transformation, mapping, synchronization, and unmapping operations. Especially for small operators, these overheads can easily become dominant, offsetting the gains of cooperative execution. Inappropriate workload partitioning: Current collaborative execution systems typically use latency predictions based on simple models (such as linear models based on FLOPs) to guide workload partitioning, but this approach has poor accuracy (e.g., μlaye's prediction accuracy is below 10%). This is because latency is not only linearly related to FLOPs but is also affected by platform characteristics (such as algorithm implementation and data block size). Inaccurate predictions lead to unbalanced workload partitioning, which in turn affects inference performance. Inaccurate predictions or complex models: Some works (such as nn-Meter) achieve high accuracy in predictions with delays by using complex black-box machine learning models, but the models have high runtime overhead and large size (e.g., the convolution operator model is over 800MB), making it impossible to make predictions in real time on mobile devices. In addition, these models do not consider concurrency-related overhead.

[0007] These issues indicate that there is still a lack of a deep learning inference method that can schedule and coordinate CPU and GPU resources under efficient and low-latency conditions. Summary of the Invention

[0008] The main objective of this invention is to provide a CPU-GPU dual-core collaborative execution method and system for mobile deep learning inference tasks, in order to solve the above-mentioned technical problems.

[0009] To achieve the above objectives, the present invention provides an optimization method for CPU-GPU co-execution of deep learning inference in mobile devices, comprising the following steps: S1, Offline Phase: Define the complete steps of operator collaborative execution, including: data transformation, mapping, pre-synchronization, computation, post-synchronization, and demapping; construct an operator delay model using a lightweight regression predictor, and calculate the total prediction delay T of operator collaborative execution; S2, Online Phase: Building a hybrid-type-friendly data sharing mechanism to allow heterogeneous processors to use different data types during inference and reduce the number of operators that need to share data, including: S21, Analyze the deep learning model and the mobile device running the deep learning model on the offline end, and perform mixed-dimensional partitioning on the tensor of the deep learning inference operator based on the given input operator settings and the partitioning plan of dimension and ratio ρ; the dimensions include three dimensions: OC, H and W. S22, link the operators of deep learning inference to form an operator chain, and perform delay calculation according to the operator delay model in step S1 to search for the balanced operator chain with minimum delay.

[0010] Preferably, the step of generating the partitioning plan with the given input operator settings and the dimensions and ratio ρ includes determining the partitioning plan for each chain head through an initial partitioning plan, the partitioning plan including the dimensions and ratio ρ, and then traversing each chain head in the order of the data flow; specifically including: S201, For the head of a chain that does not currently belong to any chain, traverse all possible ρ, and for a given ρ, perform the following search: In the order of data flow, try to add other operators to the head of the chain or form a new chain in the chain containing the head of the chain, and use a delay calculator to determine whether there is a gain before and after the addition; If there is a gain, continue adding other operators in the order of the data flow; If the gain is negative, that is, the loop condition in the pseudocode is T_gain ≥ 0, then start searching for the next partition plan with the next ρ value under the current chain head; After trying all ρ values, the best one is saved to the Chains set; S202, following the data flow order, start the process of adding different partition ratios ρ from another chain head; after processing all chain heads in the data flow order according to the above search process, the algorithm ends and finally generates a set of partition plans for a series of chains.

[0011] Preferably, step S22 includes: taking the optimal partitioning plan of each operator when there is no chain as input, and outputting the set of operator chains with the minimum total delay.

[0012] Preferably, in step S22, the data is stored in the order of the data flow. Starting with operators that are not in any chain when there is no chain, each potential ratio ρ is traversed, and the corresponding operators are linked in turn. More operators are linked continuously, and the gain compared with the case without a chain is calculated. The search stops when adding an operator no longer yields any gain. During the search process, the partition ratio with the maximum gain and the corresponding chain are recorded. After traversing all potential partition ratios, the chain with the maximum gain is added to the result set, and then the search continues for the next chain. The final operator chain is obtained.

[0013] Preferably, in step S22, each potential ratio ρ comes from operators within a search range δ around the optimal ratio ρ of the current operator, where δ is 0.3.

[0014] Preferably, in step S1, the nonlinearity caused by different algorithms and block divisions is defined, and a kernel latency prediction is constructed. The calculation formula is as follows: ;in, It is the size of the partitioned output, which is calculated for each processor from the predictor input. It is the block size given by the inference runtime for inter-core parallelism. It is the size of the basic execution unit used for intra-core parallelism, given by inference runtime or hardware parameters. It is the core quantity. It is the time required for a single execution of the basic execution unit.

[0015] Preferably, in step S1, kernel latency prediction The only learned variable in the calculation formula is It uses a very lightweight linear regression model to learn The features of this linear model are (H,W,IC,OC); where H,W,IC,OC are the height dimension, width dimension, input dimension, and output dimension, respectively.

[0016] Preferably, it also includes the following steps: S3, the CPU-GPU execution offline allocation strategy for generating the deep learning model; S4, establish the mapping relationship between deep learning models and offline allocation strategies; S5, the current mobile terminal receives a deep learning inference request, and obtains the corresponding offline allocation strategy for the current mobile terminal based on the mapping relationship in step S15 and the CPU-GPU hardware heterogeneity information of the current mobile terminal. S6, based on the current hardware load of the mobile device and the corresponding offline allocation strategy, performs pipelined parallel and asynchronous scheduling of CPU and GPU according to the preset dynamic scheduling strategy.

[0017] The present invention also provides an optimized system for CPU-GPU co-execution of deep learning inference in a mobile device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the optimized method for CPU-GPU co-execution of deep learning inference in a mobile device as described in any of the preceding claims.

[0018] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of an optimized method for CPU-GPU co-execution of deep learning inference in a mobile device as described in any of the preceding claims.

[0019] To address the shortcomings and improvement needs of existing technologies, this invention provides an optimized method and system for CPU-GPU collaborative execution of deep learning inference in mobile devices. It aims to solve the problems of high data sharing overhead, unreasonable workload partitioning, and inaccurate or excessively complex latency prediction in existing collaborative execution frameworks for deep learning inference on mobile devices. This achieves efficient CPU-GPU collaborative execution, improving inference performance, reducing energy consumption, and balancing workload. Compared with existing technologies, this invention has at least the following beneficial effects: (1) Efficient use of data types: It supports CPU and GPU to use their own efficient data types for inference, giving full play to processor performance. Compared with a cooperative execution system that uses a unified data type, it can significantly improve inference speed.

[0020] (2) Reduce data sharing overhead: By using hybrid dimension partitioning and operator chain technology, the number of data sharing times and the amount of shared data are reduced, thereby reducing the additional overhead caused by data sharing and improving collaborative execution efficiency.

[0021] (3) Accurate delay prediction: The lightweight but accurate nonlinear and concurrency-aware delay predictor can effectively take into account platform characteristics and concurrency-related overhead, achieve more reasonable workload partitioning, and accelerate the inference process.

[0022] (4) Overall performance improvement: Experiments show that compared with the most advanced existing collaborative execution system, the present invention can achieve an average speedup of 3.43 times, and up to 4.93 times, while saving 62.3% of energy, which is close to the theoretical performance limit. Attached Figure Description

[0023] The accompanying drawings, as part of this invention, are provided to further illustrate the invention. The illustrative embodiments and descriptions of the invention are used to explain the invention, but do not constitute an undue limitation thereof. Clearly, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0024] Figure 1 This is a flowchart illustrating an optimized method for CPU-GPU collaborative execution of deep learning inference in a mobile device according to an embodiment of the present invention.

[0025] Figure 2This is a schematic diagram of the hardware structure of an optimized method for CPU-GPU co-execution of deep learning inference on a mobile device, according to an embodiment of the present invention.

[0026] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0027] The technical problems solved by the embodiments of the present invention, the technical solutions adopted, and the technical effects achieved will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other equivalent or obvious variations of embodiments obtained by those skilled in the art without creative effort fall within the protection scope of the present invention. The embodiments of the present invention can be embodied in various different ways as defined and covered by the claims.

[0028] It should be noted that many specific details are given in the following description for ease of understanding. However, it is obvious that the present invention may be implemented without these specific details.

[0029] It should be noted that, in the absence of explicit limitations or conflicts, the various embodiments and their technical features in this invention can be combined with each other to form a technical solution.

[0030] μLayer and Optic enable joint CPU and GPU execution on mobile devices by using the same data types (i.e., buffer types) for both CPU and GPU to simplify data sharing. However, this design, due to the use of inefficient data types for the GPU, makes CPU+GPU execution even slower than using the GPU alone.

[0031] To directly partition operators, μLayer and Optic model operator latency using linear regression on computational cost (FLOP). While this model is lightweight, its prediction accuracy is very low (<10%) because FLOP-based predictors fail to capture actual latency behavior. Some latency predictors use complex black-box machine learning models to capture latency behavior and achieve high accuracy; however, these models incur significant runtime overhead.

[0032] The CPU-GPU co-execution design for deep learning inference on mobile devices in this invention stems from two key findings: 1) Different processors prefer different data types to achieve optimal performance. For example, we observed that for convolution computations, using image types on Adreno GPUs can achieve a 3.5× speedup compared to buffer types. For co-execution, it is necessary to use the data type that is efficient for each processor; 2) To make the latency predictor both accurate and lightweight, platform features must be incorporated into the model, rather than employing pure black-box learning.

[0033] Current performance issues of collaborative execution systems include: 1) using a uniform data type for different processors; 2) ignoring data sharing overhead; and 3) unbalanced workload partitioning.

[0034] Please refer to Figure 1 This invention proposes an optimized method for CPU-GPU co-execution of deep learning inference in mobile devices, including step S1, an offline stage; and step S2, an online stage.

[0035] S1, the offline stage, defines the complete steps of operator collaborative execution, including: data transformation, mapping, pre-synchronization, computation, post-synchronization, and demapping; it constructs an operator delay model using a lightweight regression predictor and calculates the total prediction delay T of operator collaborative execution.

[0036] Preferably, in step S1, the nonlinearity caused by different algorithms and block divisions is defined, and a kernel latency prediction is constructed. The calculation formula is as follows: ;in, It is the size of the partitioned output, which is calculated for each processor from the predictor input. It is the block size given by the inference runtime for inter-core parallelism. It is the size of the basic execution unit used for intra-core parallelism, given by inference runtime or hardware parameters. It is the core quantity. It is the time required for a single execution of the basic execution unit.

[0037] Preferably, in step S1, kernel latency prediction The only learned variable in the calculation formula is It uses a very lightweight linear regression model to learn The features of this linear model are (H,W,IC,OC); where H,W,IC,OC are the height dimension, width dimension, input dimension, and output dimension, respectively.

[0038] S2, Online Phase: Building a hybrid-type-friendly data sharing mechanism to allow heterogeneous processors to use different data types during inference and reduce the number of operators that need to share data, including: S21, Analyze the deep learning model and the mobile device running the deep learning model on the offline end, and perform mixed-dimensional partitioning on the tensor of the deep learning inference operator based on the given input operator settings and the partitioning plan of dimension and ratio ρ; the dimensions include three dimensions: OC, H and W. S22, link the operators of deep learning inference to form an operator chain, and perform delay calculation according to the operator delay model in step S1 to search for the balanced operator chain with minimum delay.

[0039] Preferably, the step of generating the partitioning plan with the given input operator settings and the dimensions and ratio ρ includes determining the partitioning plan for each chain head through an initial partitioning plan, the partitioning plan including the dimensions and ratio ρ, and then traversing each chain head in the order of the data flow; specifically including: S201, For the head of a chain that does not currently belong to any chain, traverse all possible ρ, and for a given ρ, perform the following search: In the order of data flow, try to add other operators to the head of the chain or form a new chain in the chain containing the head of the chain, and use a delay calculator to determine whether there is a gain before and after the addition; If there is a gain, continue adding other operators in the order of the data flow; If the gain is negative, that is, the loop condition in the pseudocode is T_gain ≥ 0, then start searching for the next partition plan with the next ρ value under the current chain head; After trying all ρ values, the best one is saved to the Chains set; S202, following the data flow order, start the process of adding different partition ratios ρ from another chain head; after processing all chain heads in the data flow order according to the above search process, the algorithm ends and finally generates a set of partition plans for a series of chains.

[0040] Preferably, step S22 includes: taking the optimal partitioning plan of each operator without chains as input, and outputting the set of operator chains with the minimum total latency. Specifically, the input of the algorithm is the optimal partitioning plan ParPlan of each operator without chains, stored in the data stream order, and the output chain is a set of operator chains with the minimum total latency, where the partitioning dimension, ratio ρ, and chained operator settings of each chain are correctly filled.

[0041] Preferably, in step S22, the data is stored in the order of the data flow. Starting with operators that are not in any chain when there is no chain, each potential ratio ρ is traversed, and the corresponding operators are linked in turn. More operators are linked continuously, and the gain compared with the case without a chain is calculated. The search stops when adding an operator no longer yields any gain. During the search process, the partition ratio with the maximum gain and the corresponding chain are recorded. After traversing all potential partition ratios, the chain with the maximum gain is added to the result set, and then the search continues for the next chain. The final operator chain is obtained.

[0042] Preferably, in step S22, each potential ratio ρ comes from operators within a search range δ around the optimal ratio ρ of the current operator, where δ is 0.3.

[0043] Preferably, it also includes the following steps: S3, the CPU-GPU execution offline allocation strategy for generating the deep learning model; S4, establish the mapping relationship between deep learning models and offline allocation strategies; S5, the current mobile terminal receives a deep learning inference request, and obtains the corresponding offline allocation strategy for the current mobile terminal based on the mapping relationship in step S15 and the CPU-GPU hardware heterogeneity information of the current mobile terminal. S6, based on the current hardware load of the mobile device and the corresponding offline allocation strategy, performs pipelined parallel and asynchronous scheduling of CPU and GPU according to the preset dynamic scheduling strategy.

[0044] The CPU-GPU co-execution of deep learning inference on mobile devices integrates two new technologies.

[0045] 1) Hybrid-type friendly data sharing. It allows heterogeneous processors to use different data types during inference. Then, to reduce data sharing overhead, a hybrid dimension partitioning and operator chaining approach is proposed. Hybrid dimension partitioning selects the optimal partitioning dimension for each operator shape to achieve a trade-off between data sharing overhead and processor utilization. Operator chaining ensures that operators on the chain only require local data to execute, rather than shared data from other processors, thus avoiding data sharing overhead.

[0046] 2) Nonlinear and Concurrency-Aware Latency Prediction: Online and fair operator partitioning is achieved by constructing a lightweight yet accurate latency predictor. Therefore, the nonlinear part is abstracted by formulating the computation of blocks in each algorithm through analysis; only the linear part is learned by a very lightweight model (approximately 500 bytes, compared to 800MB for nn-Meter models) via a linear regression model. Furthermore, the predictor considers all concurrency-related overhead for the first time.

[0047] Specifically, in the above embodiments, a lightweight yet effective latency predictor was designed in the offline phase to guide operator partitioning in the online phase. This predictor achieves its lightweight and effectiveness for the following reasons: 1) it considers all data sharing overhead, including data transformation, mapping, and synchronization; 2) it analytically formulates the nonlinear latency response caused by platform characteristics, and for each kernel implementation, it only requires an extremely lightweight linear regression model to learn the latency of the basic execution unit.

[0048] The role of the operator partitioner is to determine the optimal operator partitioning scheme for the input deep learning model. The final partitioning scheme of the partitioner is a set of chains found, each chain having linked operators and chain settings, i.e., partition ratio and dimension. With this scheme, the model weights are pre-arranged for GPU and CPU to avoid re-transformation on each inference call.

[0049] Based on a delay predictor, it employs two techniques—hybrid dimensionality partitioning and operator chaining—to accomplish this task. The partitioner first uses hybrid dimensionality partitioning to find the optimal partitioning dimension (height or output channels) and partitioning ratio (e.g., 0.1, 0.2, etc.) for each operator as a base scheme. Based on this base scheme, it uses operator chaining to search for operators to be linked together, ensuring that operators on the chain do not need to share data.

[0050] Figure 2 This is a schematic diagram of the hardware structure of an optimized method for CPU-GPU co-execution of deep learning inference on a mobile device, provided by an embodiment of the present invention. Figure 2 As shown, this embodiment / computer 6 includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a program for running an optimized method for CPU-GPU co-execution of deep learning inference in a mobile device. When the processor 60 executes the computer program 62, it implements the steps in the various method embodiments described above. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the various device embodiments described above.

[0051] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program 62 in the computer 6.

[0052] The computer 6 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. The computer 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 2 This is merely an example of computer 6 and does not constitute a limitation on computer 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer 6 may also include input / output devices, network access devices, buses, etc.

[0053] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0054] The memory 61 can be an internal storage unit of the computer 6, such as a hard drive or memory. The memory 61 can also be an external storage device of the computer 6, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card (FC) equipped on the computer 6. Furthermore, the memory 61 can include both internal and external storage units of the computer 6. The memory 61 is used to store the computer program 62 and other programs and data required by the computer 6. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0055] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0056] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0057] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0058] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0060] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0061] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0062] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An optimization method for CPU-GPU co-execution of deep learning inference in mobile devices, characterized in that, Including the following steps: S1, Offline Phase: Define the complete steps of operator collaborative execution, including: data transformation, mapping, pre-synchronization, computation, post-synchronization, and demapping; construct an operator delay model using a lightweight regression predictor, and calculate the total prediction delay T of operator collaborative execution; S2, Online Phase: Building a hybrid-type-friendly data sharing mechanism to allow heterogeneous processors to use different data types during inference and reduce the number of operators that need to share data, including: S21, Analyze the deep learning model and the mobile device running the deep learning model on the offline end, and perform mixed-dimensional partitioning on the tensor of the deep learning inference operator based on the given input operator settings and the partitioning plan of dimension and ratio ρ; the dimensions include three dimensions: OC, H and W. S22, link the operators of deep learning inference to form an operator chain, and perform delay calculation according to the operator delay model in step S1 to search for the balanced operator chain with minimum delay.

2. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 1, characterized in that, The steps for generating the partitioning plan with the given input operator settings and the dimensions and ratio ρ include determining the partitioning plan for each chain head through an initial partitioning plan, the partitioning plan including the dimensions and ratio ρ, and then traversing each chain head in the order of the data flow; specifically including: S201, For the head of a chain that does not currently belong to any chain, traverse all possible ρ, and for a given ρ, perform the following search: In the order of data flow, try to add other operators to the head of the chain or form a new chain in the chain containing the head of the chain, and use a delay calculator to determine whether there is a gain before and after the addition; If there is a gain, continue adding other operators in the order of the data flow; If the gain is negative, that is, the loop condition in the pseudocode is T_gain ≥ 0, then start searching for the next partition plan with the next ρ value under the current chain head; After trying all ρ values, the best one is saved to the Chains set; S202, following the data flow order, start the process of adding different partition ratios ρ from another chain head; after processing all chain heads in the data flow order according to the above search process, the algorithm ends and finally generates a set of partition plans for a series of chains.

3. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 1, characterized in that, Step S22 includes: taking the optimal partitioning plan of each operator when there is no chain as input, and outputting the set of operator chains with the minimum total delay.

4. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 3, characterized in that, In step S22, data is stored sequentially according to the data flow. Starting with operators that are not in any chain when there is no chain, each potential ratio ρ is traversed, and the corresponding operators are linked in turn. More operators are linked continuously, and the gain compared to the case without a chain is calculated. The search stops when adding an operator no longer yields any gain. During the search process, the partition ratio with the maximum gain and the corresponding chain are recorded. After traversing all potential partition ratios, the chain with the maximum gain is added to the result set, and then the search continues for the next chain. The final operator chain is obtained.

5. The optimized method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 3 or 4, characterized in that, In step S22, each potential ratio ρ comes from the operators within the search range δ around the optimal ratio ρ of the current operator, where δ is 0.

3.

6. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 1, characterized in that, In step S1, the nonlinearity caused by different algorithms and block divisions is defined, and kernel latency prediction is constructed. The calculation formula is as follows: ;in, It is the size of the partitioned output, which is calculated for each processor from the predictor input. It is the block size given by the inference runtime for inter-core parallelism. It is the size of the basic execution unit used for intra-core parallelism, given by inference runtime or hardware parameters. It is the core quantity. It is the time required for a single execution of the basic execution unit.

7. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 5, characterized in that, In step S1, kernel latency prediction The only learned variable in the calculation formula is It uses a very lightweight linear regression model to learn The features of this linear model are (H,W,IC,OC); where H,W,IC,OC are the height dimension, width dimension, input dimension, and output dimension, respectively.

8. The optimization method for CPU-GPU co-execution of deep learning inference in mobile devices according to claim 1, characterized in that, It also includes the following steps: S3, the CPU-GPU execution offline allocation strategy for generating the deep learning model; S4, establish the mapping relationship between deep learning models and offline allocation strategies; S5, the current mobile terminal receives a deep learning inference request, and obtains the offline allocation strategy corresponding to the current mobile terminal based on the mapping relationship in step S15 and the CPU-GPU hardware heterogeneity information of the current mobile terminal. S6, based on the current hardware load of the mobile device and the corresponding offline allocation strategy, performs pipelined parallel and asynchronous scheduling of CPU and GPU according to the preset dynamic scheduling strategy.

9. An optimized system for CPU-GPU co-execution of deep learning inference in a mobile device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the CPU-GPU co-execution optimization method for deep learning inference in a mobile device as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the CPU-GPU co-execution optimization method for deep learning inference in a mobile device as described in any one of claims 1 to 8.

Images