Deep learning model operation method deployed in terminal equipment system
Through category-adaptive pruning, task clustering parallelism, and heterogeneous multi-core pipeline optimization, the problem of high-precision and low-latency deployment of deep learning models on edge devices is solved, achieving efficient resource utilization and computing performance.
Patent Information
- Application Number
- CN202510512564.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-23
- Publication Date
- 2025-09-19
AI Technical Summary
On resource-constrained edge devices, the deployment of complex deep learning models faces the challenge of high-precision, low-latency real-time inference. Traditional methods lack end-to-end collaborative optimization mechanisms, resulting in problems such as reduced accuracy, response delays, or excessive energy consumption.
It adopts category-adaptive pruning, automatic model parallelism based on task clustering, and system multi-stage pipeline parallel optimization methods for heterogeneous multi-core slices to collaboratively handle model compression, parallel computing, and hardware scheduling to improve resource utilization and computing efficiency.
It significantly improves the practicality of deep learning models in real scenarios, solves the contradiction between inference accuracy and latency when resources change, and improves computing efficiency and resource utilization.
Smart Images
Figure CN120671745A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent devices based on deep learning models, and in particular to a method for operating a deep learning model deployed in a terminal device system. Background Art
[0002] Currently, the deployment of complex deep learning models in intelligent industrial and agricultural scenarios faces a systemic challenge: achieving high-precision, low-latency, real-time inference on resource-constrained edge devices. Specifically, dynamically changing operating environments (such as differences in crop growth cycles and product variations on industrial production lines) make it difficult for traditional static optimization models (such as compression models with fixed pruning rates or manually designed parallel strategies) to simultaneously meet accuracy and efficiency requirements. Furthermore, heterogeneous computing hardware (CPU / GPU / NPU hybrid architectures) suffers from low resource utilization, leading to computational and communication bottlenecks when running models in parallel.
[0003] Existing optimization methods often handle model compression, parallel computing, and hardware scheduling in isolation, lacking an end-to-end collaborative optimization mechanism. This ultimately leads to problems such as decreased accuracy, response delays, or excessive energy consumption in deployed systems in real scenarios. This core contradiction seriously restricts the practicality of smart terminal devices in areas such as agricultural automation and industrial testing. Summary of the Invention
[0004] The present invention is proposed in view of the above technical problems and provides a method for operating a deep learning model deployed in a terminal device system. By collaboratively processing model compression, parallel computing and hardware scheduling, the practicality of smart terminal devices deploying deep learning models in real scenarios is significantly improved.
[0005] According to a first aspect of an embodiment of the present invention, a method for running a deep learning model deployed in a terminal device system is provided, the method comprising:
[0006] S1. Use class-adaptive pruning methods on deep learning models to accurately reduce the size of deep learning models.
[0007] S2. Use automatic model parallelism based on task clustering for deep learning models to optimize the parallel execution strategy of deep learning models;
[0008] S3. Use a multi-stage pipeline parallel optimization method for deep learning models using heterogeneous multi-core chips to improve the inference performance and efficiency of deep learning models on heterogeneous multi-core chip systems.
[0009] According to a second aspect of an embodiment of the present invention, a method for operating a deep learning model deployed in a terminal device system as in the first aspect is provided, wherein the method includes:
[0010] S11, obtain category attention coefficient;
[0011] S12. Based on the obtained category attention coefficient, complete the category adaptive pruning of the deep learning model.
[0012] According to a third aspect of an embodiment of the present invention, a method for running a deep learning model deployed in a terminal device system as described in the second aspect is provided, wherein the method includes:
[0013] S21. Analyze operator costs based on sampled data and perform sampling analysis on the operator costs of the model, including communication cost, computation cost, and memory load cost.
[0014] S22. Construct a group edge cost model based on the task dependency clustering method, merge related operators into operator groups, and reasonably coarsen the computational graph;
[0015] S23. Use the greedy algorithm to schedule the operator group based on the communication and memory load costs in order to optimize the parallel execution strategy of the model.
[0016] According to a fourth aspect of an embodiment of the present invention, a method for running a deep learning model deployed in a terminal device system as in the first aspect is provided, wherein a method for parallel optimization of a multi-stage pipeline of a heterogeneous multi-core slice system is provided, including:
[0017] S31. Subgraph partitioning:
[0018] The deep learning model is divided into layers, and the deep learning model inference task is split into multiple subtasks. Each subtask is deployed and executed in parallel on one or more devices of the same type in the same cluster. The subtask uses a kernel-level parallel strategy to calculate the available computing resources.
[0019] S32, Buffer mechanism balances inference time:
[0020] A buffering mechanism is introduced between subgraphs. Buffers and mutexes are set between subgraphs to ensure the synchronization of pipeline execution and the correctness of data transmission.
[0021] S33, multi-stage pipeline optimization:
[0022] When performing pipeline inference, the pipeline execution order is set through parameters, and the number of threads used by the devices at each level of the pipeline is specified.
[0023] According to a fifth aspect of an embodiment of the present invention, a method for operating a deep learning model deployed in a terminal device system as in the second aspect is provided, wherein the method for obtaining a category attention coefficient includes:
[0024] First, extract M pictures from the validation set as the calibration dataset Datasets cal ={img i , i∈[1,M]}, the category of the calibration dataset is class={claass j , j∈[1,C]};
[0025] Secondly, use the N-layer model Model = {layer k , k∈[1, N]} for reasoning and calculate each picture img i Attention coefficient of the predicted category;
[0026] Finally, the attention coefficients of the same predicted category are averaged to obtain the final category attention coefficient.
[0027] According to a sixth aspect of an embodiment of the present invention, a method for running a deep learning model deployed in a terminal device system as described in the third aspect is provided, wherein the method is based on a method for analyzing operator costs based on sampled data, including:
[0028] First, the specific configuration of the device cluster and the computational graph structure of the neural network model are comprehensively considered to construct the corresponding device topology graph and model topology graph;
[0029] Secondly, based on the detailed hardware information of the device and the actual sampling data of the model operator, the execution cost of each operator is accurately calculated.
[0030] According to a seventh aspect of an embodiment of the present invention, a method for operating a deep learning model deployed in a terminal device system as described in the third aspect is provided, wherein the method for constructing a group edge cost model based on a task-dependent clustering method includes:
[0031] First, the group edge cost model is initialized by considering the dependencies between operators, and then the group edge cost is calculated;
[0032] Secondly, the task-dependency-based clustering method is used to appropriately aggregate the operators of the model to form groups.
[0033] According to an eighth aspect of an embodiment of the present invention, a method for operating a deep learning model deployed in a terminal device system as described in the third aspect is provided, wherein the method uses a greedy algorithm to implement the method, including:
[0034] First, get the initial group from the group edge model Model and place it;
[0035] Next, the algorithm enters a loop and places each group in the group edge model Model one by one until all groups have been placed.
[0036] The beneficial effects of the present invention are: by optimizing the operation of deep learning models deployed in terminal devices, collaboratively performing category adaptive pruning, automatic model parallelism based on task clustering, and multi-level pipeline parallel optimization of systems on heterogeneous multi-core chips, the collaborative optimization of model compression, parallel computing and hardware scheduling is achieved, which significantly improves the practicality of smart terminal devices that deploy deep learning models in real scenarios.
[0037] With reference to the following description and the accompanying drawings, specific embodiments of the invention are disclosed in detail, indicating the manner in which the principles of the invention may be employed.
[0038] It should be understood that the embodiments of the present invention are not limited thereby. Features described and / or shown for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or replace features in other embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings are included to provide a further understanding of the invention, and constitute a part of this specification. They illustrate preferred embodiments of the invention and together with the description serve to explain the principles of the invention. The same elements are denoted by the same reference numerals throughout.
[0040] In the attached figure:
[0041] Figure 1 This is a schematic diagram of obtaining the attention coefficient of the category in the image by taking a single image as an example for reasoning in the present invention;
[0042] Figure 2 Schematic diagram of category adaptive pruning in the present invention;
[0043] Figure 3 Schematic diagram of the calculation method of the operator computation cost, the cross-device communication cost between operators, and the memory load cost of the model in the deep learning model of the present invention;
[0044] Figure 4 This is the overall implementation flow chart of the greedy algorithm implemented in the present invention;
[0045] Figure 5 This is a flow chart of the greedy algorithm placement module in the present invention;
[0046] Figure 6 It is a schematic diagram of the multi-stage pipeline parallel optimization method in the present invention. DETAILED DESCRIPTION
[0047] With reference to the accompanying drawings, the above and other features of the present invention will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present invention are specifically disclosed, which show some embodiments in which the principles of the present invention can be adopted, and it is to be understood that the present invention is not limited to the described embodiments.
[0048] The present invention provides a method for running a deep learning model deployed in a terminal device system. First, a category adaptive pruning method is run on a single-node terminal device. The category adaptive pruning method significantly reduces the redundant parameters and computational complexity of the model, and accurately reduces the size of the deep learning model. The method adaptively adjusts the model pruning threshold according to the categories in different tasks, while considering the changes in device resources and task requirements. By establishing a resource-demand optimization model, a lightweight deep learning model that fits the current resources is obtained, and category adaptive reasoning under dynamic resources is realized, effectively solving the contradiction between reasoning accuracy and latency when the terminal device resources change. At the same time, in order to address the problems of low performance of model parallel strategies and low efficiency of strategy search, the present invention further uses an automatic model based on task clustering. Type parallel method; first, the operator cost of the model is sampled and analyzed, including communication cost, computational cost and memory load cost; next, a clustering method based on task dependency is used to construct a group edge cost model, related operators are merged into operator groups, and the computational graph is reasonably coarsened; finally, a greedy algorithm is used to schedule the operator groups with communication and memory load costs as indicators to optimize the parallel execution strategy of the model; on the basis of significantly reducing the complexity of the model and improving the computational efficiency through pruning and parallelization technology, in order to further tap the potential of hardware, the present invention is aimed at the deep optimization of heterogeneous multi-core system on chip (SoC); through fine-grained task division and heterogeneous resource collaborative scheduling, the problems of insufficient computing resource utilization and low strategy search efficiency in traditional methods are solved. Specifically, the present invention divides the deep learning model by layer, divides the deep learning model inference task into multiple subtasks, each subtask is deployed on one or more devices of the same type in the same cluster for parallel execution, and the subtask adopts a kernel-level parallel strategy to calculate the available computing resources. Because the computing resources at each pipeline stage are comprised of cores belonging to the same cluster, the overhead caused by cross-cluster data transmission is reduced. Furthermore, the present invention supports independent device assignment for each subtask through a multi-stage pipeline optimization method. Therefore, the complete pipeline can utilize all types of processor cores supported by the heterogeneous multi-core SoC, thereby improving the resource utilization of the heterogeneous multi-core SoC.
[0049] Most methods of model pruning usually only focus on the overall accuracy after pruning, but in different application scenarios, different tasks have different perceptual requirements for different categories. At the same time, for neural networks, different convolution kernels may also have different importance for different categories. Therefore, while retaining the same percentage of weight parameters, compared to pruning that considers overall accuracy, considering specific types of pruning can sacrifice the accuracy of other categories, retaining parameters that are more important to special categories, and making the accuracy of specific categories higher. Therefore, the present invention uses category attention coefficients, which are the importance of each layer to each specific category. In this way, more refined model pruning can be performed for specific categories, thereby obtaining higher performance at the same pruning rate.
[0050] According to a preferred embodiment of the present invention, the category-adaptive pruning method includes:
[0051] S11. Obtain category attention coefficient.
[0052] In order to obtain the attention coefficient of each category, we first extract M pictures from the validation set as the calibration dataset Datasets cal ={img i , i∈[1,M}}, the category of the calibration dataset is class={class j , j∈[1, C]}; Then, use the N-layer model Model={layer k , k∈[1, N]} is used to perform reasoning and calculate each image img i Finally, the attention coefficients for the same predicted category are averaged to determine the importance of each layer for each category. The importance of each layer for each category can be used to determine which weight parameters are important for a specific category, allowing for targeted pruning, more accurately reducing model size and achieving higher performance at the same pruning rate.
[0053] Figure 1 This is a schematic diagram of obtaining the attention coefficient of the category in the picture by taking a single picture for reasoning as an example in a preferred embodiment of the present invention.
[0054] Specifically, first input the image into the model for inference:
[0055] bbox,class pred =Model(img i );
[0056] Inference gets the predicted bbox and predicted category class pred After that, back propagation is performed and layerk For the predicted category class pred Gradient:
[0057]
[0058] in, The shape is B×C×H×W. At the same time, you can get layer k The output feature feat of the layer k Its shape and same;
[0059] Will Summing over the spatial dimensions:
[0060]
[0061] Next, we can get the class class pred Layer k Gradient class activation map of the layer:
[0062]
[0063] in, The shape is B×C×H×W.
[0064] According to one embodiment of the present invention, for example, for the tomato picking target detection task, the category class pred It is the category of tomatoes with different maturity of bbox, by retaining the mature category class pred In the bbox And sum it up to get the mature category class pred layer k Category attention coefficient
[0065]
[0066] When the category attention coefficients of all images in the calibration dataset are obtained When the importance coefficient of the same category as the one to be predicted is Take the average to get the final category attention coefficient
[0067] S12. Based on the obtained category attention coefficient, complete the category adaptive pruning of the deep learning model.
[0068] For a model Model that has completed sparse training, Conv={Conv k , k∈[1, C]} layer and the corresponding BN={BN k, k∈[1, C]} layers, usually, Conv k The output of the layer will be used as BN k Input;
[0069] BN k The scaling factor of the layer γ k Sort and get the set of scaling factors of the sort number γ sort ,For a given pruning rate p, the threshold of the scaling factor T can be calculated γ :
[0070] T γ =γ sort [len(γ sort )·p];
[0071] Among them, len represents the length of the collection.
[0072] At this point, we can get Conv k and BN k The index of the layer to be pruned:
[0073] mask k =γ k <T γ ;
[0074] Among them, mask k and Conv k and BN k The dimensions are the same, where the value of the element is either 0 or 1, 0 represents the layer to be removed, and 1 represents the retained layer;
[0075] Finally, the pruned Model prune Each layer is:
[0076] Conv k ′=mask k .Conv k ;
[0077] BN k ′=mask k ·BN k ;
[0078] Figure 2 This is a schematic diagram of category adaptive pruning in a preferred embodiment of the present invention. As shown in the figure, when the category attention coefficient of each layer for each category is obtained, After that, use and the scaling factor γ k To complete the pruning of a specific category, use the product of the two As an important measure of
[0079] Tα =α sort [len(α sort )·p];
[0080]
[0081] Get the mask k Then, use the formula to calculate the pruned Model prune At each layer, specific categories of pruning can be completed.
[0082] According to a preferred embodiment of the present invention, the automatic model parallel method based on task clustering includes:
[0083] S21. Analyze operator costs based on sampled data.
[0084] Figure 3 This is a schematic diagram of the calculation method for the operator computational cost, the cross-device communication cost between operators, and the memory load cost of the model of the deep learning model of the present invention. First, the specific configuration of the device cluster and the computational graph structure of the neural network model are comprehensively considered to construct the corresponding device topology diagram and model topology diagram, which facilitates the subsequent reflection of the association and mapping relationship between computing resources and model operators. Then, based on the detailed hardware information of the device and the actual sampling data of the model operator, the execution cost of each operator is accurately calculated, providing an important basis for the subsequent parallelization strategy selection and optimization.
[0085] S22. Construct a group edge cost model based on task-dependent clustering method.
[0086] According to a preferred embodiment of the present invention, the group edge cost model is first initialized by considering the dependencies between operators, and the group edge costs are then calculated. Then, a task-dependency-based clustering method is used to appropriately aggregate the model's operators into groups, narrowing the search space. This helps improve the efficiency of the optimization algorithm, reduces computational costs, and increases the likelihood of finding a more optimal solution. This comprehensive task-dependency modeling and clustering strategy provides an effective and flexible approach for optimizing deep learning models.
[0087] S23. Greedy algorithm implementation.
[0088] The greedy algorithm of the present invention takes communication as the primary benchmark to ensure that the model is placed in a way that minimizes communication; secondly, it takes device capacity as the secondary benchmark to reduce the probability of memory overflow during the placement process; finally, it determines whether the aggregation group is completely traversed to confirm that all operators have obtained the corresponding strategy. Figure 4This is the overall implementation flow chart of the greedy algorithm of the present invention. As shown in the figure, the initial group is first obtained from the group edge model Model and placed. This is the initial stage; then, the algorithm enters a loop and places each group in the group edge model Model one by one; if the group has not been visited, it will enter the greedy algorithm placement module; Figure 5 This is the flow chart of the greedy algorithm placement module of the present invention, see Figure 5 In this module, when selecting a placement device for the current group, the device that is connected to the current group and has the largest communication volume already placed is first considered as the preferred pre-placement device, and the capacity threshold of this pre-placement device is evaluated. If placing the current group on this device would exceed its capacity threshold, the next best communication option is considered one by one as an alternative pre-placement device. If all alternative devices fail to meet the capacity threshold, the device with the largest capacity in the current device cluster is selected as the final pre-placement device. To ensure that all groups have been placed and prevent errors, when the group placement is completed and the greedy algorithm placement module is exited, it is necessary to check whether all groups in the model have been placed. If there are still unplaced groups, the unplaced groups are obtained and the loop is entered again until all groups have been placed.
[0089] According to a preferred embodiment of the present invention, the method for parallel optimization of a system multi-stage pipeline of a heterogeneous multi-core slice includes:
[0090] S31. Subgraph partitioning:
[0091] The deep learning model is divided into layers, and the deep learning model inference task is split into multiple subtasks. Each subtask is deployed on one or more devices of the same type in the same cluster for parallel execution. The subtask uses a kernel-level parallel strategy to calculate the available computing resources.
[0092] The design of the pipeline inference solution includes two parts: model segmentation and device partitioning. The reasoning process of the model is essentially the sequential execution of various operators, which can be abstracted as executing each node in a topological order. The present invention will add splitting points between these nodes to perform model segmentation. Since the convolutional layer and the fully connected layer account for most of the total inference time, and the other layers account for only a small part. Therefore, when performing model segmentation, the present invention only considers setting model splitting points for the convolutional layer and the fully connected layer, and merges the other layers with the convolutional layer or the fully connected layer, which is called a layer. For a deep learning model containing P layers, it can be divided into a maximum of P subgraphs (that is, a split point is inserted after each layer), and a minimum of 1 subgraph (that is, no split point is inserted).
[0093] This invention divides the node sequence of the original deep learning model into subsequences called subgraphs. Each subgraph can be independently configured, such as the backend type and number of threads. To ensure accurate data transmission between subgraphs, this invention inserts sending and receiving nodes between subgraphs. Specifically, for the first subgraph of a deep learning model, its receiving nodes are equivalent to the input nodes of the original model; for the last subgraph, its sending nodes are equivalent to the output nodes of the original model.
[0094] S32, Buffer mechanism balances inference time:
[0095] Since the unbalanced workload of each subgraph will lead to differences in the inference time of each level of the pipeline, the present invention introduces a buffer mechanism between subgraphs, sets up buffers and mutex locks between subgraphs to ensure the synchronization of pipeline execution and the correctness of data transmission.
[0096] After the current subgraph completes the calculation, the output data will be stored in the buffer for the next subgraph to receive. The sending node first locks the buffer through the mutex mechanism to ensure that the subsequent operations are thread-safe, and then checks whether the buffer is in a waiting state. If it is not in a waiting state, it means that the receiving node is not ready to receive data, or there is unread data in the buffer. At this time, the sending node will store the Tensor to be sent in the cache queue so that the receiving node can use it later; if the buffer is in a waiting state, it means that the receiving node is ready to receive data and the buffer is empty. At this time, the sending node will mark the data state as ready to inform the receiving node that the data is ready. After completing the operation, the buffer is unlocked to allow other threads to access the buffer.
[0097] When the next subgraph completes the calculation, the receiving node will first access the buffer to read the new input data, and then check whether the buffer queue is empty. If the buffer queue is not empty, the data in the buffer queue will be read sequentially to the input Tensor of the next subgraph until the buffer queue is cleared, which means that the data has been successfully transmitted from the sending node to the receiving node; if the buffer queue is empty, the receiving node will set the buffer state to the waiting state, and the current thread (receiving node) will be in a blocked state until the sending node inserts data into the buffer, marks the data state as ready and unlocks it. At this time, the receiving node will be released from the waiting state and obtain data from the buffer.
[0098] S33, multi-stage pipeline optimization:
[0099] To achieve fine-grained partitioning of devices, when performing pipeline inference, it supports setting the pipeline execution order through parameters and specifying the number of threads for the devices used at each level of the pipeline. The default number of threads for each device is 1. When the number of threads specified for a device is greater than 1, multi-threading is enabled, the workload on the current device is divided, and the workload is allocated to other devices in the same cluster for parallel operation through device affinity binding. Therefore, for a cluster containing N devices, it can be divided into a maximum of N levels of pipelines (each level of pipeline contains 1 device), and a minimum of 1 level of pipeline (N threads are enabled so that N devices can jointly support the inference of 1 level of pipeline). Figure 6 This is a schematic diagram of the multi-stage pipeline parallel optimization method in the present invention. This method avoids the cross-cluster data transmission overhead problem encountered in traditional kernel-level parallel strategies, and significantly improves the inference performance and efficiency of deep learning models on heterogeneous multi-core system-on-chips through hierarchical model segmentation, buffer queue mechanism, data transmission control and fine-grained device division.
[0100] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, and many features and advantages of these embodiments are apparent from this detailed description. In addition, since many modifications and changes will readily occur to those skilled in the art, it is not intended that the embodiments of the present invention be limited to the exact construction and operation illustrated and described, but rather that all suitable modifications and equivalents are encompassed.
Claims
1. A method for operating a deep learning model deployed in a terminal device system, characterized in that: The method comprises: S1. Use class-adaptive pruning methods on deep learning models to accurately reduce the size of deep learning models. S2. Use automatic model parallelism based on task clustering for deep learning models to optimize the parallel execution strategy of deep learning models; S3. Use a multi-stage pipeline parallel optimization method for deep learning models using heterogeneous multi-core chips to improve the inference performance and efficiency of deep learning models on heterogeneous multi-core chip systems.
2. A method for operating a deep learning model deployed in a terminal device system according to claim 1, wherein: The category-adaptive pruning method includes: S11, obtain category attention coefficient; S12. Based on the obtained category attention coefficient, complete the category adaptive pruning of the deep learning model.
3. A method for operating a deep learning model deployed in a terminal device system according to claim 1, wherein: The automatic model parallel method based on task clustering includes: S21. Analyze operator costs based on sampled data and perform sampling analysis on the operator costs of the model, including communication cost, computation cost, and memory load cost. S22. Construct a group edge cost model based on the task-dependent clustering method, merge related operators into operator groups, and reasonably coarsen the computational graph; S23. Use the greedy algorithm to schedule the operator group based on the communication and memory load costs in order to optimize the parallel execution strategy of the model.
4. A method for operating a deep learning model deployed in a terminal device system according to claim 1, wherein: The method for parallel optimization of a system multi-stage pipeline for a heterogeneous multi-core chip includes: S31, subgraph division: The deep learning model is divided into layers, and the deep learning model inference task is split into multiple subtasks. Each subtask is deployed and executed in parallel on one or more devices of the same type in the same cluster. The subtask uses a kernel-level parallel strategy to calculate the available computing resources. S32, Buffer mechanism balances inference time: A buffering mechanism is introduced between subgraphs. Buffers and mutexes are set between subgraphs to ensure the synchronization of pipeline execution and the correctness of data transmission. S33, multi-stage pipeline optimization: When performing pipeline inference, the pipeline execution order is set through parameters, and the number of threads used by the devices at each level of the pipeline is specified.
5. A method for operating a deep learning model deployed in a terminal device system according to claim 2, wherein: The method for obtaining the category attention coefficient includes: First, extract M pictures from the validation set as the calibration dataset Datasets cal ={img i , i∈[1,M]}, the category of the calibration data set is class={class j , j∈[1,C]}; Secondly, use the N-layer model Model = {layer k , k∈[1, N]} is used to perform reasoning and calculate each image img i Attention coefficient of the predicted category; Finally, the attention coefficients of the same predicted category are averaged to obtain the final category attention coefficient.
6. A method for operating a deep learning model deployed in a terminal device system according to claim 3, wherein: The method for analyzing operator costs based on sampled data includes: First, the specific configuration of the device cluster and the computational graph structure of the neural network model are comprehensively considered to construct the corresponding device topology graph and model topology graph; Secondly, based on the detailed hardware information of the device and the actual sampling data of the model operator, the execution cost of each operator is accurately calculated.
7. A method for operating a deep learning model deployed in a terminal device system according to claim 3, wherein: The method for constructing a group edge cost model based on a task-dependent clustering method includes: First, the group edge cost model is initialized by considering the dependencies between operators, and then the group edge cost is calculated; Secondly, the task-dependency-based clustering method is used to appropriately aggregate the operators of the model to form groups.
8. A method for operating a deep learning model deployed in a terminal device system according to claim 3, wherein: The implementation method of using the greedy algorithm includes: First, get the initial group from the group edge model Model and place it; Next, the algorithm enters a loop and places each group in the group edge model Model one by one until all groups have been placed.