Terminal and edge server cooperative model inference optimization method and system, and storage medium

Abstract

The application provides a terminal and edge server cooperative model reasoning optimization method and system and a storage medium. The method comprises the following steps: acquiring the computing power parameters and communication parameters of local terminals and edge servers in an initial model deployment environment; combining the computing power parameters, the communication parameters and layer structure information of the initial model to calculate a total splitting cost; taking the total splitting cost and model reasoning accuracy as optimization objectives, processing the initial model based on an optimal pruning strategy to generate a lightweight model; selecting a target splitting point matched with a real-time communication rate from the matching relationship between the communication rate interval and the splitting point, splitting the lightweight model into a terminal-side sub-model and an edge-side sub-model based on the target splitting point, deploying the terminal-side sub-model on the local terminal and deploying the edge-side sub-model on the edge server to carry out cooperative reasoning. The embodiment of the application can ensure reasoning accuracy while realizing low delay.

Description

Technical Field

[0001] This application relates to the field of wireless communication technology, specifically to a model inference optimization method, system, and storage medium for collaboration between a terminal and an edge server. Background Technology

[0002] With the improvement of computing power in mobile terminals, local deployment of AI inference on mobile devices has become feasible. However, deep neural networks have a large number of parameters, and high latency issues arise when the terminal performs inference alone. Traditional cloud-based inference solutions require long-distance transmission of raw data, resulting in high communication overhead and inference lag caused by network jitter. Existing optimization paths fall into two categories: model lightweighting and terminal-edge collaborative inference. Simple model pruning can significantly reduce recognition accuracy, while conventional collaborative inference uses fixed model splitting points, which cannot adapt to dynamically fluctuating wireless communication rates, making it difficult to balance inference accuracy and real-time performance.

[0003] Existing collaborative reasoning optimization schemes have significant shortcomings: First, they do not cluster and model historical network speed data, but only use a single fixed bandwidth to calculate the split latency, which cannot quantify the comprehensive reasoning cost under variable network conditions. Second, model pruning, task splitting, and communication resource allocation are optimized independently, lacking a linkage mechanism. The pruning strategy is not dynamically updated in conjunction with network conditions, and the model splitting position is not adaptively adjusted according to real-time predicted network speed. Furthermore, communication resource blocks are not dynamically allocated for multiple terminals, ultimately making it difficult for resource-constrained terminals to stably achieve low-latency, high-precision AI collaborative reasoning. Summary of the Invention

[0004] In view of this, this application proposes a model inference optimization method, system and storage medium for terminal and edge server collaboration, in order to solve the problem that existing terminal-edge collaborative inference schemes cannot simultaneously guarantee inference accuracy and low latency.

[0005] The first aspect of this application proposes a model inference optimization method for collaboration between a terminal and an edge server, the method comprising: Obtain the computing power parameters and communication parameters of the local terminal and the edge server in the initial model deployment environment. The computing power parameters are used to characterize the computing performance of the local terminal and the edge server, and the communication parameters are used to characterize the historical communication rate distribution between the local terminal and the edge server. Combining the computing power parameters, the communication parameters, and the layer structure information of the initial model, calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server; the execution cost includes computing latency and transmission latency. Based on the execution cost of each computing layer, all computing layers in the initial model are traversed, and the total inference latency corresponding to the complete inference of the initial model when each computing layer is taken as the model splitting point is calculated. The total inference latency is determined as the splitting cost corresponding to the model splitting point. Clustering is performed based on the historical communication rate distribution to obtain multiple central transmission rates. The model splitting point corresponding to the minimum splitting cost under each central transmission rate is determined as a candidate splitting point. The total splitting cost of the initial model is obtained by weighted summation based on the splitting costs corresponding to multiple candidate splitting points. With the overall splitting cost and model inference accuracy as optimization objectives, the pruning rate of each computational layer is determined by reinforcement learning, and the optimal pruning strategy that satisfies the accuracy constraint is output. The initial model is then processed based on the optimal pruning strategy to generate a lightweight model. Target splitting points that match the real-time communication rate are selected from the pre-established matching relationship between communication rate ranges and splitting points. Based on the target splitting points, the lightweight model is split into a terminal-side sub-model and an edge-side sub-model. The terminal-side sub-model is deployed on the local terminal, and the edge-side sub-model is deployed on the edge server to carry out collaborative inference.

[0006] An embodiment of the second aspect of this application provides a model inference optimization system that coordinates terminal and edge server, including: The parameter acquisition module is used to acquire the computing power parameters and communication parameters of the local terminal and the edge server in the initial model deployment environment. The computing power parameters are used to characterize the computing performance of the local terminal and the edge server, and the communication parameters are used to characterize the historical communication rate distribution between the local terminal and the edge server. An execution cost calculation module is used to calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server, by combining the computing power parameters, the communication parameters, and the layer structure information of the initial model; the execution cost includes computing latency and transmission latency; The splitting cost determination module is used to traverse all computing layers in the initial model based on the execution cost of each computing layer, calculate the total inference latency corresponding to the complete inference of the initial model when each computing layer is taken as the model splitting point, and determine the total inference latency as the splitting cost corresponding to the model splitting point. The overall splitting cost calculation module is used to cluster the historical communication rate distribution to obtain multiple central transmission rates, determine the model splitting point corresponding to the minimum splitting cost under each central transmission rate as a candidate splitting point, and obtain the overall splitting cost of the initial model by weighted summation of the splitting costs corresponding to multiple candidate splitting points. The lightweight model generation module is used to optimize the overall splitting cost and model inference accuracy by using reinforcement learning to determine the pruning rate of each computational layer, outputting the optimal pruning strategy that satisfies the accuracy constraint, and processing the initial model based on the optimal pruning strategy to generate a lightweight model. The model deployment module is used to filter out target split points that match the real-time communication rate from the pre-established matching relationship between communication rate ranges and split points. Based on the target split points, the lightweight model is split into terminal-side sub-models and edge-side sub-models. The terminal-side sub-models are deployed on the local terminal, and the edge-side sub-models are deployed on the edge server to carry out collaborative inference.

[0007] An embodiment of the third aspect of this application provides a computer-readable storage medium storing computer instructions for causing a computer to execute the model inference optimization method for terminal-edge server collaboration described in the first aspect.

[0008] The beneficial effects of this application are: (1) It can solve the splitting point decision problem of different models. A unified splitting cost quantification model is established, and the potential splitting point of the model is optimized in the pruning stage. In the task splitting stage, the optimal splitting point is automatically matched for different models through the rate interval matching mechanism, so that the splitting decision has hierarchical flexibility and transferability; (2) It can solve the model pruning problem in multi-objective optimization. By using the reinforcement learning reward function, accuracy and latency are unified in the optimization objective, providing a model foundation for achieving accurate and timely inference under edge-end collaboration. Moreover, the pruning strategy will change with the historical wireless communication rate distribution to cope with the dynamic network environment.

[0009] (3) It can improve the utilization efficiency of computing and communication resources. By jointly considering the model splitting location, wireless communication status and resource allocation, the system can dynamically adjust the task execution mode and communication resource allocation strategy according to the network environment, so that the terminal side computing power, edge side computing power and wireless communication resources can be used in a coordinated manner, thereby reducing the performance loss caused by resource idleness and resource competition, and improving the overall execution efficiency of collaborative reasoning tasks and the system resource utilization rate. Attached Figure Description

[0010] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1This illustration shows a flowchart of a model inference optimization method for terminal-edge server collaboration provided in an embodiment of this application; Figure 2 This illustration shows a flowchart of obtaining a lightweight model through an optimal pruning strategy according to an embodiment of this application; Figure 3 This paper illustrates a flowchart of another model inference optimization method for terminal-edge server collaboration provided in an embodiment of this application. Figure 4 This illustration shows a schematic diagram of the structure of a model inference optimization system that integrates terminal and edge server according to an embodiment of this application; Figure 5 This illustration shows a structural schematic diagram of a high-efficiency collaborative reasoning optimization device for AI tasks provided in an embodiment of this application; Figure 6 A schematic diagram of a storage medium provided in one embodiment of this application is shown. Detailed Implementation

[0011] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0012] It should be noted that, unless otherwise stated, the technical or scientific terms used in this application shall have the ordinary meaning as understood by one of ordinary skill in the art to which this application pertains.

[0013] The technical scenarios involved in the embodiments of this application are described below.

[0014] In recent years, the increasing computing power-to-volume ratio of mobile hardware has made it possible to implement AI inference services on mobile devices. However, the complexity of AI models is also constantly increasing, and timely and reliable inference requires powerful computing capabilities. For complex deep neural networks, the number of parameters can reach hundreds of millions, making it difficult for mobile devices to independently and promptly perform the corresponding inference tasks. In traditional cloud-centric solutions, large amounts of data need to be transmitted bidirectionally between the mobile device and a remote cloud data center over long distances via wide area networks, which can lead to unacceptable latency and huge communication overhead.

[0015] To achieve timely and reliable inference on resource-constrained devices, current research focuses on two main directions: firstly, optimizing network size to reduce the overall number of model parameters; and secondly, splitting the network model across distributed terminals and edge servers to collaboratively complete inference tasks. However, simply reducing model size often comes at the cost of inference accuracy, and existing collaborative inference mechanisms are still not perfect in the face of complex and ever-changing real-world deployment environments. Currently, achieving low latency while maintaining inference accuracy remains a pressing issue, especially under heterogeneous inference models and dynamic network conditions, where the design of collaborative inference schemes is particularly crucial. Therefore, a highly efficient collaborative inference optimization method is urgently needed to reduce the inference latency of AI tasks while ensuring inference accuracy, enabling resource-constrained terminal devices to execute large-scale AI models and improving the accessibility of intelligent services.

[0016] According to an embodiment of this application, a model inference optimization method embodiment for collaboration between a terminal and an edge server is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0017] Example 1: Limited by the computing power and battery capacity of mobile devices, relying solely on local computing is insufficient for efficiently handling highly complex tasks such as object detection and semantic segmentation. Optimizing network size can reduce the overall number of model parameters, but it reduces model accuracy. Collaborative inference based on model decomposition can utilize multi-level computing resources, but inefficient task offloading strategies may exacerbate latency. Currently, achieving low latency while ensuring inference accuracy remains a pressing issue, especially under conditions of heterogeneous model structures and dynamic networks, where the design of collaborative inference schemes is particularly critical. To address this, this invention proposes a model inference optimization method that integrates terminal and edge server collaboration. This method can reduce inference latency for AI tasks while maintaining inference accuracy, thereby achieving efficient collaboration between communication and computing resources. This enables resource-constrained terminal devices to execute large-scale AI models, improving the overall performance of edge intelligent systems.

[0018] Figure 1 This is a flowchart of a model inference optimization method for terminal and edge server collaboration according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain the computing power parameters and communication parameters of the local terminal and edge server in the initial model deployment environment.

[0019] Specifically, computing power parameters are used to characterize the computing capabilities of each computing node (e.g., local terminal, edge server). For example, the computing power of each computing node can be characterized by the number of floating-point operations per second (FLOPs). Communication parameters are used to characterize the long-term historical communication rate distribution between the local terminal and the edge server. The computing power parameters and communication parameters can be measured during system deployment, statistically analyzed during operation and maintenance, or provided by the network operator, and their specific values ​​are not limited.

[0020] More specifically, in the following text, the computing power of the local terminal can be expressed as: An edge server can be represented as The communication rate between the two can be expressed as .

[0021] Step S102: Combining the computing power parameters, the communication parameters, and the layer structure information of the initial model, calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server.

[0022] Specifically, the hierarchical structure of the initial model is analyzed to extract the layer structure information; the initial model is... , This represents the total number of computational layers in the initial model. Indicates the first Layers; the extracted layer structure information includes, but is not limited to, any computational layer in the initial model. computational load Size of intermediate feature data Potential structural constraints (e.g., whether residual connections are included); where computational cost represents the computational overhead required for the forward propagation of the corresponding computational layer, and intermediate feature data size is used to characterize the dimension and data volume of the output feature map of the corresponding computational layer, and is used to estimate the transmission load.

[0023] In some specific embodiments, if the first The layer output feature map size is Then the size of the intermediate feature data output by this computational layer can be expressed as: ,in, , Used to characterize the first The number of channels in each computational layer, the pixel height of the output feature map, and the pixel width of the output feature map; This indicates the number of data bits occupied by a single feature element.

[0024] In some specific embodiments, step S102 includes steps S1021-S1022: Step S1021: For any computing layer, determine the computing latency of the local terminal based on the computing load of the computing layer and the computing power of the local terminal, and determine the computing latency of the edge server based on the computing load of the computing layer and the computing power of the edge server.

[0025] Step S1022: For any historical communication rate in the historical communication rate distribution, determine the transmission delay between the local terminal and the edge server based on the historical communication rate and the amount of output feature data of the computing layer.

[0026] In this embodiment, the execution cost of each computing layer is calculated across different execution nodes (local terminal or edge server). The execution cost includes computation latency and transmission latency. Computation latency is determined by the ratio of the computational load of each layer to the computing power of the corresponding execution node, reflecting the time required to execute that layer on a specified node. Specifically, it is divided into computation latency for the local terminal and computation latency for the edge server. Transmission latency is determined by the size of the intermediate feature data output by each layer and the transmission rate between the local terminal and the edge server, describing the latency overhead incurred during cross-node transmission.

[0027] Step S103: Based on the execution cost of each computing layer, traverse all computing layers in the initial model, calculate the total inference latency corresponding to the complete inference of the initial model when each computing layer is used as a model splitting point, and determine the total inference latency as the splitting cost corresponding to the model splitting point.

[0028] In some specific embodiments, step S103 above includes steps S1031-S1032: Step S1031: For any computing layer among all the computing layers, under the condition that the computing layer is used as the model splitting point, the computing terminal computing latency, edge computing latency, and model splitting point transmission latency are calculated.

[0029] Specifically, the terminal computing latency is the sum of the computing latencies of all computing layers before the model splitting point on the local terminal, the edge computing latency is the sum of the computing latencies of all computing layers after the model splitting point on the edge server, and the model splitting point transmission latency is the transmission latency of the computing layer that serves as the model splitting point.

[0030] Step S1032: Based on the terminal computing latency, the edge computing latency, and the model splitting point transmission latency, calculate the total inference latency under the condition that the computing layer is used as the model splitting point.

[0031] In this embodiment of the application, under the condition that any computing layer is used as a model splitting point, the terminal computing latency, edge computing latency, and model splitting point transmission latency can be calculated in the following way:

[0032] in, Indicates when the first Terminal computation latency when a layer is used as a model splitting point. Indicates the first The computational cost of each layer This represents the computing power of the local terminal; the formula above indicates that the local terminal is responsible for processing the data from the previous... Each computational layer processes the data, and the previous... The sum of the computing latency of each computing layer running on the local terminal is used as the terminal computing latency.

[0033]

[0034] in, Indicates when the first Edge computation latency when a layer is used as a model splitting point. Indicates the first The computational cost of each layer This represents the computing power of the edge server; the formula above indicates that the edge server is responsible for processing the data from the front-end... The computational layer processes the data, and the first layer processes the data. To the last computational layer The sum of the computation latency of each running on the edge server is taken as the edge computing latency.

[0035]

[0036] in, Indicates the transmission delay at the model split point. Indicates the first The size of the intermediate feature data output by each computational layer. This indicates the current communication rate between the local terminal and the edge server; the formula above means that the local terminal will... The latency of transmitting intermediate feature data (i.e. feature maps) output by each computation layer to the edge server.

[0037] More specifically, the total inference latency (split cost) is the sum of the terminal computing latency, the edge computing latency, and the model splitting point transmission latency, as shown below:

[0038] Since the inference results of the model are usually very small, the latency of the inference results being returned is ignored.

[0039] In some specific embodiments, under fixed computing power parameters, for any communication rate from a local terminal to the edge server, a layer with the lowest splitting cost can be obtained, as shown below:

[0040] in, Indicates the communication rate as The optimal split point at time, variables This is used to illustrate that the optimal split point changes with real-time network speed; the optimal split layer varies with different network speeds. Indicates the first The layer is the split point, and the communication rate is... Total latency (split cost) of complete edge-to-edge collaborative inference at any time.

[0041] This layer represents the model at the corresponding transmission rate. The optimal split point is given by the minimum split cost.

[0042] In this embodiment of the application, when the hardware computing power of the local terminal and the edge server is fixed, an arbitrary wireless transmission rate is given. Each of these methods can uniquely determine the optimal split point for a layer, as well as the corresponding minimum inference latency; subsequently, all layers can be weighted based on the historical communication rate distribution. The overall cost of the calculated model split is used for reinforcement learning pruning optimization.

[0043] Step S104: Cluster the historical communication rate distribution to obtain multiple central transmission rates, determine the model splitting point corresponding to the minimum splitting cost under each central transmission rate as a candidate splitting point, and perform a weighted summation based on the splitting costs corresponding to multiple candidate splitting points to obtain the overall splitting cost of the initial model.

[0044] In some specific embodiments, step S104 above includes steps S1041-S1044: Step S1041: Cluster all historical communication rates in the historical communication rate distribution to obtain multiple groups; each group corresponds to a central transmission rate, and each group includes several historical communication rates.

[0045] Specifically, it can be based on the historical communication rate distribution. Using K-means clustering, we obtain Cluster centers (i.e., the groups obtained from the above clustering process), for example Each cluster center corresponds to a sample set. Each sample set includes several historical communication rates.

[0046] Step S1042: For any group, calculate the sample percentage corresponding to the group based on the number of rate samples contained in the group and the total number of historical communication rates.

[0047] Specifically, as shown in the following formula:

[0048] in, Indicates the first The sample percentage of each group Indicates the first The number of rate samples contained in each group. This represents the total number of historical communication rates included in the historical communication rate distribution.

[0049] Step S1043: For any central transmission rate, using the central transmission rate as a fixed transmission rate, determine the splitting cost corresponding to each computing layer in the initial model as a model splitting point, and determine the model splitting point corresponding to the minimum splitting cost as a candidate splitting point.

[0050] Specifically, the current central transmission rate can be... As a variable of fixed transmission rate Substitute all computational layers of the initial model Split cost function In the middle; traverse all layer splitting points and calculate the inference latency corresponding to each layer. By minimizing the solution, the globally minimum splitting cost at the central transmission rate is obtained. and the corresponding model splitting points The split point of this model is the candidate split point for matching in this group; its mathematical expression is as follows:

[0051]

[0052] Step S1044: Using the sample proportion weights corresponding to each group as weighting coefficients, the minimum splitting cost determined by the transmission rate of each center is weighted and summed to obtain the overall splitting cost of the initial model.

[0053] Specifically, based on the sample proportion corresponding to each group. As a weighting factor, the minimum splitting cost corresponding to the central rate of each group. The weighted summation is performed; the overall breakdown cost calculation formula is shown below:

[0054]

[0055] Among them, the calculated Used to characterize the overall performance of the model in collaborative inference at different transmission rates, and to serve as a reference for the objective function of subsequent optimization, in order to measure the overall performance of pruning operations and split point selection.

[0056] This application embodiment compresses massive historical communication rate samples through clustering, and obtains the overall splitting cost by weighting the optimal splitting cost of each scenario with the proportion of grouped samples as the weight. While preserving the wireless network rate distribution characteristics, it significantly reduces the amount of offline computation. At the same time, the optimal splitting latency under each network speed scenario is used for weighting, so that the overall splitting cost can represent the long-term average inference overhead of the system. This serves as the pruning optimization objective, allowing the lightweight model to have balanced low latency performance in various network scenarios.

[0057] Step S105: Taking the overall splitting cost and model inference accuracy as optimization objectives, the pruning rate of each computational layer is determined by reinforcement learning, the optimal pruning strategy that satisfies the accuracy constraint is output, and the initial model is processed based on the optimal pruning strategy to generate a lightweight model.

[0058] Specifically, the implementation process for generating a lightweight model based on the optimal pruning strategy for the initial model is as follows: Figure 2 As shown, the "simplified model" in the process refers to the lightweight model in the embodiments of this application. For example... Figure 3 The “S1 model pruning” is shown in the figure.

[0059] More specifically, this step can be based on reinforcement learning methods, employing an agent to make layer-by-layer pruning rate decisions for the model, treating model pruning as a continuous action decision-making process layer by layer. Specifically, the agent selects an appropriate pruning rate for each layer, gradually optimizing the model's accuracy and splitting cost through iterative learning. Let the initial model have... In the layer, the reinforcement learning agent is in the first layer. The observation state corresponding to the layer can be represented as ,in, Indicates the index of the current layer to be pruned. Indicates the computational load of the current layer. Indicates the number of channels in the current layer. Indicates the dimension of the output features of the current layer. Indicates the current layer type. This represents the overall cost of splitting the current model. Correspondingly, the action output by the agent is defined as the pruning rate of the current layer. This is used to indicate the retention ratio of the current layer weight channel.

[0060] In some specific embodiments, step S105 above includes steps S1051-S1056: Step S1051: In the current training round, the pruning agent is invoked to traverse from the first layer to the last layer of the initial model, read the observation state of the current layer, and output the pruning rate action corresponding to the current layer according to the observation state.

[0061] Specifically, initialize the current traversed layer index to the first layer of the model, and start the current complete training round; read the observation state corresponding to the current layer. The observed state is input into the initialized / updated pruning agent policy network, and the network outputs the pruning rate action corresponding to the current layer. , The parameter is Pruning agent policy network; pruning rate The value range is limited to 0 to 1. The pruning rate represents the proportion of channels that need to be pruned in this layer. Record the current layer's observation status and the output pruning rate action. The temporary cache is used to store training experience later. If the current layer is not the last layer of the model, jump to the next layer and update the observation status, and repeat this step. If it is the last layer, then enter S1052.

[0062] More specifically, for example: The model has 12 layers, and this round starts from the first layer; the state of the third layer is read (convolutional layer, 256 channels, 7 GFLOPs of computation, and the current total splitting cost is 33ms). The agent outputs a pruning rate of 0.6, which means that 60% of the redundant channels in this layer are pruned, and 40% of the effective channels are retained.

[0063] Step S1052: Prune the current layer channel according to the current layer pruning rate, update the overall structure of the initial model, and recalculate the overall splitting cost of the pruned model.

[0064] Specifically, based on the pruning rate of the current layer's output, equivalent structured pruning can be performed on the convolutional / fully connected channels of that layer: sorting by channel weights, removing the corresponding proportionally redundant channels with the lowest weight magnitude, and simultaneously deleting the corresponding output-side connected channels; updating the complete network structure, number of channels in each layer, and amount of output feature data for each layer of the current temporary model in real time; calling the clustering weighting process described above, recalculating the overall splitting cost corresponding to the current model based on the updated model layer parameters, and updating this real-time overall splitting cost to the observation state for use by the next layer's agent in decision-making.

[0065] Step S1053: If all layers of the initial model have been traversed, the inference accuracy of the pruned model is determined, and the reward value is calculated based on the inference accuracy and the overall splitting cost of the model before and after pruning. The reward value of this round, the observation state of each layer, and the pruning action are stored as training experience in the experience replay pool.

[0066] Specifically, if the current layer is the last layer of the model, then this round ends, and the reward value is calculated according to the reward function. The reward function is expressed as:

[0067] in, This indicates the inference accuracy of the pruned model on the validation dataset. This represents the original total cost of the model. This represents the total splitting cost of the model after pruning. Used to adjust the importance of accuracy in rewards. This is used to adjust the range of the reward function. The reward function comprehensively considers the improvement in splitting cost while maintaining high model accuracy. When the model maintains high accuracy and splitting cost is reduced, the reward value is higher; conversely, when pruning leads to a significant decrease in accuracy or the improvement in splitting performance is not significant, the reward value is lower. This mechanism enables the model to minimize the cost of cross-node inference while maintaining accuracy.

[0068] In some specific embodiments, the inference accuracy of the pruned model is evaluated on a relevant dataset. Different accuracy evaluation methods can be selected based on the structural characteristics of different models.

[0069] Specifically, for DNN models with close inter-layer dependencies and strict model structure constraints, such as ResNet, after determining the model pruning rate, a lightweight model that conforms to the pruning rate is generated, and its accuracy is verified after a short training period. For models with relatively loose inter-layer dependencies and relatively loose model structure constraints, such as VGG, the pruning channel is determined by orthogonal matching pursuit, an equivalent pruning method using masks is adopted, and the least squares method is used for weight regression before verifying the model accuracy.

[0070] In the application embodiment, after calculating the reward value, the training experience of this round is stored in the experience replay pool.

[0071] Step S1054: Periodically obtain training experience from the experience replay pool, use the training experience to update the policy parameters of the pruning agent, and use the updated pruning agent for the next training round.

[0072] Specifically, during the training process, a portion of samples can be periodically drawn from the experience replay pool to train the agent and update the agent's network parameters.

[0073] Step S1055: When the training rounds of the pruning agent reach the preset maximum training rounds, traverse all training rounds and select the layer-by-layer pruning scheme with inference accuracy greater than the preset accuracy threshold and the lowest overall splitting cost as the optimal pruning strategy.

[0074] Specifically, as the training process continues, the reinforcement learning agent gradually learns the optimal pruning strategy, enabling the model to minimize partitioning costs while maintaining accuracy. The system then determines whether the maximum number of training rounds has been reached; if so, training ends.

[0075] More specifically, a two-layer screening constraint is applied to all pruning schemes: the first layer of screening: all unqualified pruning schemes with inference accuracy lower than the preset accuracy threshold are eliminated; the second layer of screening: among all remaining qualified schemes, the layer-by-layer pruning scheme with the smallest overall splitting cost is selected, and then the set of exclusive pruning rates for each layer recorded in the scheme is determined as the optimal pruning strategy for the model.

[0076] For example, with a maximum training run of 500 rounds, a total of 500 pruning schemes are saved. After screening, 327 schemes with an accuracy of ≥88% remain. Among them, the scheme in round 362 has the lowest overall splitting cost (22.7ms). The combination of pruning rates for each layer in this scheme is the optimal pruning strategy.

[0077] Step S1056: Process the initial model according to the optimal pruning strategy to generate a lightweight model.

[0078] Specifically, the target pruning rate corresponding to each layer in the optimal pruning strategy is read, and the original unpruned initial model is loaded; structured channel pruning is performed on the initial model layer by layer in order, deleting low-weight redundant channels in each layer and reconstructing a simplified network topology; the pruned network is then subjected to short-term fine-tuning training: the model weights are iteratively updated using a small number of samples from the training set to repair the accuracy loss caused by channel pruning and stabilize the model's inference accuracy; the simplified network after fine-tuning is solidified and saved to obtain the final lightweight model.

[0079] This application employs a layer-by-layer sequential reinforcement learning pruning approach. After each layer of pruning, the model structure and overall splitting cost are updated in real time, enabling the agent to perceive the latency changes brought about by the lightweighting of previous layers and achieving a dynamic balance of pruning intensity between layers. The strategy optimization is guided by a reward function that integrates accuracy and latency, and the agent is iteratively updated by experience replay. After multiple rounds of training, the pruning scheme that meets the accuracy standard and has the optimal latency is selected. Finally, a lightweight model is output, which reduces the amount of feature data transmitted across devices from the source, and forms a complete lightweight generation closed loop that can be directly used for edge-device splitting and deployment.

[0080] Step S106: Select target splitting points that match the real-time communication rate from the pre-established matching relationship between communication rate ranges and splitting points. Based on the target splitting points, split the lightweight model into terminal-side sub-models and edge-side sub-models. Deploy the terminal-side sub-models on the local terminal and the edge-side sub-models on the edge server to carry out collaborative inference.

[0081] Specifically, such as Figure 3 As shown in “S2 Task Splitting”, the lightweight model is split into a terminal-side sub-model and an edge-side sub-model based on the target splitting point. The terminal-side sub-model is then deployed on the local terminal, and the edge-side sub-model is deployed on the edge server.

[0082] In some specific embodiments, step S106 above includes steps S1061-S1066: Step S1061: Using different network layers as splitting points and the wireless communication rate as a variable, calculate the expression for the splitting cost of each splitting point in terms of the wireless communication rate.

[0083] Specifically, the fixed hardware parameters of the initial model are known: local terminal computing power. Edge server computing power The computational cost of each layer of the lightweight model has been statically analyzed. The output feature map size of each layer can be used to calculate the amount of feature data transmitted at each layer. .

[0084] Static analysis was performed on the computational cost of the initial model, the output data size of each layer, and the inter-layer dependencies. Different network layers were used as candidate splitting points to measure the wireless communication rate. As variables, the expression for the splitting cost with respect to the wireless communication rate at each splitting point is calculated, as shown below:

[0085] Among them, local terminal computing power Edge server computing power The computational cost of each layer of the lightweight model has been statically analyzed. The output feature map size of each layer can be used to calculate the amount of feature data transmitted at each layer. .

[0086] Step S1062: Calculate the critical network speed threshold based on the expression corresponding to any two split points.

[0087] Specifically, two different sets of candidate split points can be arbitrarily selected. , This makes the cost of splitting the two equal. Then, substitute the corresponding expressions to find the unique positive real solution. The solution is the critical network speed threshold where the time delay of the two split points is equal. Finally, traverse all pairwise split point combinations, repeatedly solve the simultaneous solution, and collect all valid critical network speed thresholds. Remove negative numbers and infinite solutions that have no actual physical meaning, and only retain valid thresholds greater than 0.

[0088] Step S1063: Sort all critical network speed thresholds and divide them into continuous communication rate intervals.

[0089] Specifically, all effective critical network speed thresholds are sorted in ascending order from smallest to largest. Based on the sorted thresholds, the entire wireless rate is divided into intervals, generating multiple continuous, non-overlapping, and complete rate intervals. The interval format is uniformly left-closed and right-open, with the smallest interval covering extremely weak networks and the largest interval covering high-speed wireless networks. For example, effective thresholds: 3.2Mbps and 8.89Mbps, divided into three intervals: Interval 1: 0≤r<3.2Mbps; Interval 2: 3.2≤r<8.89Mbps; Interval 3: r≥8.89Mbps.

[0090] Step S1064: Select the optimal split point with the lowest splitting cost for each interval and establish a matching relationship between the communication rate interval and the optimal split point.

[0091] Specifically, for each segmented communication rate interval, an arbitrary representative test rate is selected within the interval; this test rate is substituted into the cost expression of all split points to calculate the split cost corresponding to each split point; the split cost values ​​of all split points within the current interval are compared, and the split point with the smallest value is selected as the exclusive optimal split point for that interval; each rate interval is bound to its corresponding optimal split point, generating a mapping table and persistently storing it on the edge server and the local terminal to form a matching relationship; the matching relationship satisfies the constraint that when any real-time rate falls into a certain interval, the split cost of the split point bound to that interval is strictly less than all other split points.

[0092] For example, if the interval 0 ≤ r < 3.2 is tested at a rate of 2 Mbps, C9 is calculated to be the minimum, and the optimal split point is bound to the 9th layer; if the interval 3.2 ≤ r < 8.89 is bound to the 6th layer; if the interval r ≥ 8.89 is bound to the 3rd layer, a matching table is generated and stored.

[0093] In some specific embodiments, after calculating the expression for the splitting cost of each splitting point with respect to the wireless communication rate, the method includes: identifying potential optimal splitting points in the model with the aim of minimizing the splitting cost, and establishing a matching relationship between the splitting points and the communication rate. Here, the matching relationship refers to the communication rate range corresponding to each splitting point; when the actual communication rate is within this range, the splitting cost of the corresponding splitting point is less than the splitting cost of other splitting points.

[0094] Specifically, for any two candidate split points and The system compares the size relationships between corresponding split cost functions. Solve for the range of wireless communication rates that satisfy the above conditions, and thus obtain the split point. Corresponding effective communication rate range Iterate through all candidate split points and calculate their corresponding effective communication rate ranges. When the rate range corresponding to a certain split point satisfies... If the split point has the minimum splitting cost under at least one communication rate condition, then the system adds it to the set of potential optimal split points. Furthermore, the system records the communication rate range corresponding to each potential optimal split point. ,in, and They represent the splitting points respectively. This corresponds to the upper and lower bounds of the communication rate range. The split point is defined when the actual wireless communication rate falls within this range. The corresponding splitting cost is lower than other candidate splitting points. Through the above process, a matching relationship between splitting points and communication rate ranges is established, enabling the optimal splitting point under the current communication conditions to be quickly determined without having to retrace all splitting points.

[0095] Step S1065: Predict the real-time communication rate based on the current channel state and the historical communication rate distribution.

[0096] Specifically, a predictor based on a long short-term memory network can be used to predict the real-time communication rate that can be achieved at present, based on the current channel state and the historical communication rate distribution.

[0097] In this embodiment of the application, the historical communication rate distribution prediction is assumed to be... ,in, Indicates the length of the historical observation window. Obtains current wireless channel state information. The historical communication rate distribution and the current channel state are input together into the Long Short Time Memory network predictor to obtain the predicted wireless communication rate. ,in, This represents a prediction model for long short-term memory networks.

[0098] In some specific embodiments, the Long Short-Term Memory (LSTM) network can be trained by minimizing the mean squared error between the predicted rate and the actual rate, and its loss function is expressed as:

[0099] in, This represents the input sequence.

[0100] Step S1066: Determine the target split point that matches the real-time communication rate based on the matching relationship.

[0101] Specifically, the predicted wireless communication rate With the set of potential optimal split points The communication rate ranges corresponding to each split point are matched. When the condition is met... Determine the current target split point as .

[0102] This application embodiment solves the critical network speed by splitting the cost function to divide the continuous rate interval, selects the optimal split point in each interval to establish a matching mapping relationship, and obtains the target split point online by only combining the channel state and historical rate to predict the real-time network speed and then looking up the table. There is no need to solve complex time delay equations in real time, which reduces the computing power overhead of online scheduling. At the same time, it relies on strict mathematical division to ensure that the splitting scheme within the interval is globally optimal. The weak network automatically splits deeply to reduce the transmission volume, and the high-speed network splits shallowly to offload the calculation, realizing the adaptive dynamic model splitting of the wireless network.

[0103] In some specific embodiments, the lightweight model is split into a terminal-side sub-model and an edge-side sub-model based on the target splitting point. The terminal-side sub-model is deployed on the local terminal, and the edge-side sub-model is deployed on the edge server, including: Based on the determined target split point The initial model is split into a terminal-side sub-model and an edge-side sub-model, where the first layer to the second layer... The layer will be executed on the local terminal device, and the intermediate feature data will be uploaded to the edge server to execute the next layer. Layer to the first The computation layer of the layer. In this layer, the local terminal device performs forward inference computation of the terminal-side sub-model and prepares to upload intermediate feature data to the edge server; during this process, it counts the size of the intermediate feature data and the task latency requirements to provide a basis for the allocation of communication resources in the subsequent transmission process.

[0104] By using the above method, it is not necessary to recalculate all splitting cost functions at runtime. The optimal splitting point under the current communication conditions can be quickly determined by simply matching the predicted communication rate with the pre-established communication rate range, thereby reducing the complexity of online decision-making and improving the real-time performance of collaborative reasoning.

[0105] In some specific embodiments, the method further includes: Step a1: Based on the size of intermediate feature data of multiple local terminals, task latency requirements, current channel status, task priority, and resource block occupancy status, the current resource allocation status is determined.

[0106] Specifically, at the beginning of each task cycle, a resource allocation scheme is generated for each terminal based on the size of the intermediate feature data to be transmitted and the task latency requirements, combined with the current wireless channel state information, resource block (RB) occupancy, and task priority. The resource allocation scheme can be trained using reinforcement learning, enabling the system to gradually learn resource block allocation strategies based on historical network states and task execution results; or it can be dynamically allocated using rule-based optimization, heuristic search, or traditional optimization algorithms.

[0107] More specifically, such as Figure 3 As shown in the “S3 Communication Resource Block Allocation”, there are a total of Each local terminal has a resource block set. ,in, Indicates the number of available resource blocks. For the terminal The target split points can be obtained as follows: The size of the intermediate feature data that needs to be uploaded is The task latency requirement is expressed as Based on the above information, a resource allocation scheme is generated for the terminal.

[0108] Step a2: Generate a resource allocation scheme based on the current resource allocation status.

[0109] Specifically, according to each terminal intermediate feature data size Task latency requirements Current channel state Information such as resource block occupancy status and terminal task priority constitutes the current resource allocation status. ,in, This represents the set of resource blocks that are currently occupied or unavailable. The system generates a resource allocation scheme based on the current state. ,in, Indicates terminal At any moment Is a resource block allocated? , can be represented as ,when When, it indicates a resource block. Assigned to terminal ;when When the value is "not allocated", it indicates that no allocation has been made. Resource allocation schemes can be generated using reinforcement learning methods, rule optimization, heuristic search, greedy allocation, traditional optimization algorithms, etc. This invention does not limit the specific resource allocation algorithm. Its goal is to minimize intermediate feature transmission latency and improve resource utilization while meeting task latency requirements and resource constraints.

[0110] Step a3: Allocate corresponding resource blocks to each local terminal according to the resource allocation scheme, so as to transmit intermediate feature data from the corresponding local terminal to the corresponding edge server.

[0111] This application's embodiments construct a multi-dimensional resource allocation state-coordinated scheduling wireless resource block based on lightweight and adaptive splitting. It can cope with wireless resource competition conflicts in multi-terminal concurrent transmission scenarios, and is compatible with various allocation algorithms such as reinforcement learning, greedy algorithms, and heuristics to adapt to edge devices with different computing power. By reasonably allocating communication resources, it improves the actual transmission bandwidth of the terminal, further compresses feature upload latency from the wireless transmission level, and builds a three-layer collaborative optimization system of model, splitting, and communication resources.

[0112] In some specific embodiments, the method further includes steps b1-b3: Step b1: For any local terminal, calculate the total coordination inference latency of the local terminal based on the local computing latency of the local terminal, the edge computing latency of the corresponding edge server, and the transmission latency between the local terminal and the corresponding edge server.

[0113] Specifically, the transmission latency and actual transmission rate of each terminal under the current resource allocation scheme are calculated, and the total collaborative inference latency is obtained by combining the local computing latency and the edge computing latency.

[0114] More specifically, after receiving complete intermediate feature data, the edge server performs forward inference computation on the edge sub-model and returns the inference results to each terminal device. This is combined with local computation latency. and edge computing latency and transmission latency Get terminal Total delay of collaborative inference .

[0115] Step b2: Determine whether the total coordination inference latency meets the task latency requirements of the local terminal.

[0116] Specifically, determine whether the task meets the delay constraint. If the conditions are met, the task is considered to be able to be completed on time under the current resource allocation scheme; if not, the resource allocation strategy needs to be adjusted in subsequent cycles.

[0117] Step b3: If the total coordination inference latency does not meet the task latency requirements of the local terminal, the resource allocation scheme is adjusted according to the resource allocation feedback information. The resource allocation feedback information includes the total coordination inference latency, transmission latency, resource block utilization, task completion status, and latency constraint satisfaction status of each local terminal.

[0118] Specifically, after each task cycle, the total collaborative inference latency, transmission latency, resource block utilization, task completion status, and latency constraint satisfaction of each terminal are statistically analyzed, and resource allocation feedback information is generated. Subsequent resource allocation strategies are then adjusted based on this feedback information. For example, when a terminal has tight task latency or large intermediate feature data, its resource allocation priority can be increased; when a terminal has poor channel quality, more resource blocks can be allocated to it, or it can wait for a more suitable transmission opportunity; when resource block contention is strong, coordinated allocation can be performed by combining task priority, remaining latency budget, and channel conditions.

[0119] In embodiments employing reinforcement learning for resource allocation, the current network state, resource allocation actions, and task execution results can be used as training samples. A reward signal can be constructed based on total task latency, resource utilization, and task completion status to progressively optimize the resource allocation strategy. In embodiments employing non-reinforcement learning for resource allocation, rule parameters, priority weights, or heuristic search strategies can be dynamically updated based on task execution feedback.

[0120] This application embodiment collects multi-dimensional feedback information such as total collaborative latency, resource utilization, and latency compliance after resource allocation is executed. For terminals that fail to meet latency standards, the resource allocation strategy is dynamically adjusted. Resources can be allocated differently based on terminal channel, data volume, and task priority to form a closed-loop self-optimization mechanism for resource scheduling cycle, continuously improving wireless resource utilization and latency compliance rate of high-priority tasks.

[0121] In some specific embodiments, the historical wireless communication rate distribution of each terminal is statistically analyzed, and the pruning strategy and model deployment are updated periodically. During continuous system operation, the system records the actual wireless communication rate of each terminal at different times. Based on the changes in the historical wireless communication rate distribution of each terminal, the system periodically determines whether step S101 needs to be re-executed. When a significant change in the wireless communication environment is detected, or when a preset update cycle is reached, the system re-statistically analyzes the historical rate distribution and updates the overall splitting cost of the model based on the new communication rate distribution. Subsequently, the system retrains or updates the pruning strategy, generating a new pruned model, and deploys the updated terminal-side sub-model and edge-side sub-model to the terminal device and edge server, respectively. Through this method, the system can adaptively update the model structure and collaborative inference configuration as the wireless environment changes, thereby continuously reducing inference latency and improving communication-computing resource utilization efficiency during long-term operation.

[0122] Example 2: Corresponding to the above implementation of the model inference optimization method for terminal-edge server collaboration, this application also provides a model inference optimization system for terminal-edge server collaboration, used to execute the model inference optimization method for terminal-edge server collaboration described in the above embodiments. Figure 4 As shown, the model inference optimization system that coordinates the terminal and the edge server includes: The parameter acquisition module is used to acquire the computing power parameters and communication parameters of the local terminal and the edge server in the initial model deployment environment. The computing power parameters are used to characterize the computing performance of the local terminal and the edge server, and the communication parameters are used to characterize the historical communication rate distribution between the local terminal and the edge server. An execution cost calculation module is used to calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server, by combining the computing power parameters, the communication parameters, and the layer structure information of the initial model; the execution cost includes computing latency and transmission latency; The splitting cost determination module is used to traverse all computing layers in the initial model based on the execution cost of each computing layer, calculate the total inference latency corresponding to the complete inference of the initial model when each computing layer is taken as the model splitting point, and determine the total inference latency as the splitting cost corresponding to the model splitting point. The overall splitting cost calculation module is used to cluster the historical communication rate distribution to obtain multiple central transmission rates, determine the model splitting point corresponding to the minimum splitting cost under each central transmission rate as a candidate splitting point, and obtain the overall splitting cost of the initial model by weighted summation of the splitting costs corresponding to multiple candidate splitting points. The lightweight model generation module is used to optimize the overall splitting cost and model inference accuracy by using reinforcement learning to determine the pruning rate of each computational layer, outputting the optimal pruning strategy that satisfies the accuracy constraint, and processing the initial model based on the optimal pruning strategy to generate a lightweight model. The model deployment module is used to filter out target split points that match the real-time communication rate from the pre-established matching relationship between communication rate ranges and split points. Based on the target split points, the lightweight model is split into terminal-side sub-models and edge-side sub-models. The terminal-side sub-models are deployed on the local terminal, and the edge-side sub-models are deployed on the edge server to carry out collaborative inference.

[0123] Optionally, the execution cost calculation module is further configured to, for any computing layer, determine the computing latency of the local terminal based on the computing load of the computing layer and the computing power of the local terminal, and determine the computing latency of the edge server based on the computing load of the computing layer and the computing power of the edge server; and for any historical communication rate in the historical communication rate distribution, determine the transmission latency between the local terminal and the edge server based on the historical communication rate and the amount of output feature data of the computing layer.

[0124] Optionally, the splitting cost determination module is further configured to, for any computing layer among all the computing layers, calculate the terminal computing latency, edge computing latency, and model splitting point transmission latency, under the condition that the computing layer is used as the model splitting point; wherein, the terminal computing latency is the sum of the computing latency of all computing layers before the model splitting point on the local terminal, the edge computing latency is the sum of the computing latency of all computing layers after the model splitting point on the edge server, and the model splitting point transmission latency is the transmission latency of the computing layer used as the model splitting point; based on the terminal computing latency, the edge computing latency, and the model splitting point transmission latency, calculate the total inference latency under the condition that the computing layer is used as the model splitting point.

[0125] Optionally, the overall splitting cost calculation module is further configured to cluster all historical communication rates in the historical communication rate distribution to obtain multiple groups; each group corresponds to a central transmission rate, and each group includes several historical communication rates; for any group, the sample proportion corresponding to the group is calculated based on the number of rate samples contained in the group and the total number of historical communication rates; for any central transmission rate, the splitting cost corresponding to each computational layer in the initial model is determined as the model splitting point using the central transmission rate as a fixed transmission rate, and the model splitting point corresponding to the minimum splitting cost is determined as a candidate splitting point; the minimum splitting cost determined by each central transmission rate is weighted and summed using the sample proportion weight of each group as a weighting coefficient to obtain the overall splitting cost of the initial model.

[0126] Optionally, the lightweight model generation module is further configured to, in the current training round, invoke the pruning agent to traverse the initial model from the first layer to the last layer sequentially, read the observation state of the current layer, and output the pruning rate action corresponding to the current layer based on the observation state; prune the current layer channel according to the current layer pruning rate, update the overall structure of the initial model, and recalculate the overall splitting cost of the pruned model; if all layers of the initial model have been traversed, determine the inference accuracy of the pruned model, and calculate the reward value based on the inference accuracy and the overall splitting cost of the model before and after pruning. The reward value of this round, the observation state of each layer, and the pruning actions are stored as training experience in the experience replay pool. Training experience is periodically retrieved from the experience replay pool, and the policy parameters of the pruning agent are updated using the training experience. The updated pruning agent is then used in the next training round. When the training rounds of the pruning agent reach the preset maximum training rounds, all training rounds are iterated through, and the layer-by-layer pruning scheme with inference accuracy greater than a preset accuracy threshold and the lowest overall splitting cost is selected as the optimal pruning strategy. The initial model is then processed according to the optimal pruning strategy to generate a lightweight model.

[0127] Optionally, the model deployment module is further configured to: use different network layers as splitting points and wireless communication rate as a variable; calculate the expression for the splitting cost of each splitting point in terms of wireless communication rate; calculate the critical network speed threshold based on the expression for any two splitting points; sort all critical network speed thresholds and divide them into continuous communication rate intervals; filter the optimal splitting point with the minimum splitting cost for each interval and establish a matching relationship between the communication rate interval and the optimal splitting point; predict the real-time communication rate based on the current channel state and the historical communication rate distribution; and determine the target splitting point that matches the real-time communication rate based on the matching relationship.

[0128] Optionally, the system further includes: a resource allocation module, used to construct a current resource allocation state based on the size of intermediate feature data of multiple local terminals, task latency requirements, current channel status, task priority, and resource block occupancy status; generate a resource allocation scheme based on the current resource allocation state; and allocate corresponding resource blocks to each local terminal according to the resource allocation scheme, so as to perform the step of transmitting intermediate feature data from the corresponding local terminal to the corresponding edge server.

[0129] Optionally, the system further includes: a resource allocation adjustment module, used to calculate the total coordinated inference latency of any local terminal based on the local computing latency of the local terminal, the edge computing latency of the corresponding edge server, and the transmission latency between the local terminal and the corresponding edge server; determine whether the total coordinated inference latency meets the task latency requirements of the local terminal; if the total coordinated inference latency does not meet the task latency requirements of the local terminal, adjust the resource allocation scheme according to resource allocation feedback information; the resource allocation feedback information includes the total coordinated inference latency, transmission latency, resource block utilization, task completion status, and latency constraint satisfaction status of each local terminal.

[0130] The terminal-edge server collaborative model inference optimization system provided in the above embodiments of this application and the terminal-edge server collaborative model inference optimization method provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0131] Example 3: This invention proposes a high-efficiency collaborative reasoning optimization device for AI tasks, such as... Figure 5 As shown. The device includes: an edge server 100 and at least one terminal device 200.

[0132] The edge server 100 is connected to the terminal device 200 via a wireless communication link, and is used to realize model optimization, model deployment, task splitting, wireless resource allocation, and edge-end collaborative inference.

[0133] The edge server 100 is used to perform model optimization, model management, communication resource allocation, and edge-side inference. In one specific embodiment, the edge server 100 can be an edge computing node composed of one or more high-performance computing servers, and its hardware may include a processor, a graphics processor, memory, a communication interface, and computing units for performing deep neural network inference and model training. At the software level, the edge server 100 includes a model optimization module 101, a model management module 102, a communication resource allocation module 103, an information management and monitoring module 104, and an edge inference execution module 105.

[0134] The model optimization module 101 is used to perform structural analysis on the target deep neural network, extract the computational cost of network layers, the size of output feature data, and inter-layer dependencies, and establish a mapping relationship between model splitting cost and network layers. Further, the model optimization module 101 is used to perform layer-by-layer structured pruning training on the deep neural network based on reinforcement learning methods to obtain the pruning strategy with the minimum splitting cost under accuracy constraints. The model optimization module 101 is also used to reconstruct a lightweight model according to the optimal pruning strategy and to fine-tune or retrain the pruned model.

[0135] The model management module 102 is used to store the original deep neural network model, the pruned lightweight model, and the corresponding model version information. Further, the model management module 102 is used to update the model version based on the pruning results output by the model optimization module 101, and to send the corresponding lightweight model or model configuration to the terminal device 200, enabling the terminal device 200 to execute corresponding terminal-side inference tasks.

[0136] The communication resource allocation module 103 is used to dynamically generate a communication resource allocation scheme for one or more terminal devices 200 based on the task requirement information, intermediate feature data size, wireless channel status information, and resource block occupancy information uploaded by the terminal devices 200. Furthermore, the communication resource allocation module 103 is used to adjust the resource block allocation strategy according to the resource allocation result to reduce the transmission latency of the intermediate feature data.

[0137] The information management and monitoring module 104 is used to collect statistics on wireless communication rate information, channel status information, resource usage information, and task execution logs. Furthermore, the information management and monitoring module 104 provides historical wireless communication rate distribution, resource usage status, and task execution feedback to the model optimization module 101 and the communication resource allocation module 103 to support pruning strategy updates and resource allocation strategy adjustments.

[0138] The edge inference execution module 105 is used to receive intermediate feature data uploaded by the terminal device 200, load the edge sub-model provided by the model management module 102, and perform forward inference calculations of the network layer after the split point to generate the final inference result. Further, the edge inference execution module 105 can return the final inference result to the terminal device 200 or send it to the corresponding business application module.

[0139] The terminal device 200 is used to perform task splitting decision-making, local inference, intermediate feature uploading, and wireless status acquisition. In one specific embodiment, the terminal device 200 can be a smartphone, drone, vehicle terminal, camera, wearable device, industrial sensing device, or other resource-constrained smart terminal. Its hardware may include an embedded processor, neural network processing unit, memory, wireless communication module, and sensor module. At the software level, the terminal device 200 includes a model storage module 201, a status acquisition and control module 202, a task splitting module 203, a local inference execution module 204, and a data transmission module 205.

[0140] The model storage module 201 is used to store the lightweight model and model splitting related configurations issued by the edge server 100. For example, the model storage module 201 can store pruned model files, model hierarchy information, and candidate splitting point information.

[0141] The status acquisition and control module 202 is used to acquire task requirement information, wireless channel status information, wireless communication rate information, and model splitting execution status of the terminal device 200, and upload them to the edge server 100. Further, the status acquisition and control module 202 is used to receive model update information and resource allocation configuration issued by the edge server 100, and control the terminal device 200 to execute the corresponding collaborative inference process.

[0142] The task splitting module 203 is used to predict the possible wireless communication rate based on the current wireless channel state information and historical wireless communication rate sequences, and to determine the optimal splitting point under the current communication conditions based on a pre-established communication rate interval and splitting point matching relationship. Further, the task splitting module 203 is used to generate a model splitting configuration based on the determined optimal splitting point, instructing the terminal device 200 to execute the network layer before the splitting point, and handing over the network layer after the splitting point to the edge server 100 for execution.

[0143] The local inference execution module 204 is used to load the corresponding terminal-side sub-model from the model storage module 201 according to the model splitting configuration generated by the task splitting module 203, and perform forward inference calculation of the terminal-side sub-model to generate intermediate feature data.

[0144] The data transmission module 205 is used to upload the intermediate feature data generated by the local inference execution module 204 to the edge server 100 according to the resource allocation scheme issued by the communication resource allocation module 103 of the edge server 100. Further, the data transmission module 205 is also used to receive the final inference result returned by the edge server 100 and feed back the transmission status to the status acquisition and control module 202.

[0145] Through the above device structure, the edge server 100 can be responsible for computationally intensive model optimization, model management, resource allocation, and edge-side inference. The terminal device 200 can autonomously complete task splitting, local inference, and intermediate feature uploading based on the real-time communication status, thereby achieving collaborative optimization between model pruning, task splitting, and communication resource allocation, reducing the latency of edge-end collaborative inference of AI tasks while ensuring inference accuracy.

[0146] This application also provides a computer-readable storage medium corresponding to the terminal-edge server collaborative model inference optimization method provided in the foregoing embodiments. Please refer to... Figure 6 The computer-readable storage medium shown is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it executes the model inference optimization method for terminal and edge server collaboration provided in any of the foregoing embodiments.

[0147] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0148] The computer-readable storage medium provided in the above embodiments of this application and the model reasoning optimization method for terminal and edge server collaboration provided in the embodiments of this application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the applications stored therein.

[0149] It should be noted that: Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known structures and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0150] Similarly, it should be understood that, for the sake of brevity and to aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this disclosure should not be construed as reflecting a schematic diagram in which the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0151] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0152] The above description is merely a preferred embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model inference optimization method for collaboration between terminals and edge servers, characterized in that, The method includes: Obtain the computing power parameters and communication parameters of the local terminal and the edge server in the initial model deployment environment. The computing power parameters are used to characterize the computing performance of the local terminal and the edge server, and the communication parameters are used to characterize the historical communication rate distribution between the local terminal and the edge server. Combining the computing power parameters, the communication parameters, and the layer structure information of the initial model, calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server; the execution cost includes computing latency and transmission latency. Based on the execution cost of each computing layer, all computing layers in the initial model are traversed, and the total inference latency corresponding to the complete inference of the initial model when each computing layer is taken as the model splitting point is calculated. The total inference latency is determined as the splitting cost corresponding to the model splitting point. Clustering is performed based on the historical communication rate distribution to obtain multiple central transmission rates. The model splitting point corresponding to the minimum splitting cost under each central transmission rate is determined as a candidate splitting point. The total splitting cost of the initial model is obtained by weighted summation based on the splitting costs corresponding to multiple candidate splitting points. With the overall splitting cost and model inference accuracy as optimization objectives, the pruning rate of each computational layer is determined by reinforcement learning, and the optimal pruning strategy that satisfies the accuracy constraint is output. The initial model is then processed based on the optimal pruning strategy to generate a lightweight model. Target splitting points that match the real-time communication rate are selected from the pre-established matching relationship between communication rate ranges and splitting points. Based on the target splitting points, the lightweight model is split into a terminal-side sub-model and an edge-side sub-model. The terminal-side sub-model is deployed on the local terminal, and the edge-side sub-model is deployed on the edge server to carry out collaborative inference.

2. The method according to claim 1, characterized in that, Combining the computing power parameters, the communication parameters, and the layer structure information of the initial model, the execution cost of each computing layer in the initial model when running on the local terminal or the edge server is calculated, including: For any computing layer, the computing latency of the local terminal is determined based on the computing load of the computing layer and the computing power of the local terminal, and the computing latency of the edge server is determined based on the computing load of the computing layer and the computing power of the edge server. For any historical communication rate in the historical communication rate distribution, the transmission delay between the local terminal and the edge server is determined based on the historical communication rate and the amount of output characteristic data of the computing layer.

3. The method according to claim 1 or 2, characterized in that, Based on the execution cost of each computation layer, all computation layers in the initial model are traversed, and the total inference latency corresponding to the complete inference of the initial model when each computation layer is used as a model splitting point is calculated, including: For any computing layer among all the computing layers, under the condition that the computing layer is used as the model splitting point, the computing terminal computing latency, edge computing latency, and model splitting point transmission latency are defined as follows: the terminal computing latency is the sum of the computing latencies of all computing layers before the model splitting point on the local terminal; the edge computing latency is the sum of the computing latencies of all computing layers after the model splitting point on the edge server; and the model splitting point transmission latency is the transmission latency of the computing layer used as the model splitting point. Based on the terminal computing latency, the edge computing latency, and the model splitting point transmission latency, the total inference latency is calculated under the condition that the computing layer is used as the model splitting point.

4. The method according to claim 1, characterized in that, Clustering is performed based on the historical communication rate distribution to obtain multiple central transmission rates. The model splitting point corresponding to the minimum splitting cost under each central transmission rate is determined as a candidate splitting point. The total splitting cost of the initial model is obtained by weighted summation of the splitting costs corresponding to multiple candidate splitting points, including: All historical communication rates in the historical communication rate distribution are clustered to obtain multiple groups; each group corresponds to a central transmission rate, and each group includes several historical communication rates; For any given group, calculate the sample percentage corresponding to that group based on the number of rate samples contained in the group and the total number of historical communication rates; For any central transmission rate, using the central transmission rate as a fixed transmission rate, determine the splitting cost corresponding to each computing layer in the initial model when it is used as a model splitting point, and determine the model splitting point corresponding to the minimum splitting cost as a candidate splitting point; Using the sample proportion weights corresponding to each group as weighting coefficients, the minimum splitting cost determined by the transmission rate of each center is weighted and summed to obtain the overall splitting cost of the initial model.

5. The method according to claim 1, characterized in that, With overall splitting cost and model inference accuracy as optimization objectives, reinforcement learning is used to determine the pruning rate of each computational layer, outputting an optimal pruning strategy that satisfies accuracy constraints. Based on this optimal pruning strategy, the initial model is processed to generate a lightweight model, including: In the current training round, the pruning agent is invoked to traverse from the first layer to the last layer of the initial model, read the observation state of the current layer, and output the pruning rate action corresponding to the current layer according to the observation state. The current layer channel is pruned according to the current layer pruning rate, the overall structure of the initial model is updated, and the overall splitting cost of the pruned model is recalculated. If all layers of the initial model have been traversed, the inference accuracy of the pruned model is determined, and the reward value is calculated based on the inference accuracy and the overall splitting cost of the model before and after pruning. The reward value of this round, the observation state of each layer, and the pruning action are stored as training experience in the experience replay pool. Training experience is periodically retrieved from the experience replay pool, and the training experience is used to update the policy parameters of the pruning agent. The updated pruning agent is then used in the next training round. When the training rounds of the pruning agent reach the preset maximum training rounds, the training results of all rounds are traversed, and the layer-by-layer pruning scheme with inference accuracy greater than the preset accuracy threshold and the minimum overall splitting cost is selected as the optimal pruning strategy. The initial model is processed according to the optimal pruning strategy to generate a lightweight model.

6. The method according to claim 1, characterized in that, Target split points that match the real-time communication rate are selected from the pre-established matching relationship between communication rate ranges and split points, including: Using different network layers as splitting points and wireless communication rate as a variable, we calculate the expression for the splitting cost of each splitting point in terms of wireless communication rate. Calculate the critical network speed threshold based on the expression corresponding to any two split points; Sort all critical network speed thresholds and divide them into continuous communication rate intervals; The optimal split point with the lowest splitting cost is selected interval by interval, and a matching relationship between the communication rate interval and the optimal split point is established. Predict the real-time communication rate based on the current channel state and the historical communication rate distribution; Based on the matching relationship, a target split point that matches the real-time communication rate is determined.

7. The method according to claim 1 or 2, characterized in that, The method further includes: The current resource allocation status is determined based on the size of intermediate feature data from multiple local terminals, task latency requirements, current channel status, task priority, and resource block occupancy status. Generate a resource allocation scheme based on the current resource allocation status; According to the resource allocation scheme, each local terminal is allocated a corresponding resource block to carry out the step of transmitting intermediate feature data from the corresponding local terminal to the corresponding edge server.

8. The method according to claim 7, characterized in that, After allocating corresponding resource blocks to each local terminal according to the resource allocation scheme to enable the corresponding local terminal to transmit intermediate feature data to the corresponding edge server, the method further includes: For any local terminal, the total coordination inference latency of the local terminal is calculated based on the local computing latency of the local terminal, the edge computing latency of the corresponding edge server, and the transmission latency between the local terminal and the corresponding edge server. Determine whether the total coordination inference latency meets the task latency requirements of the local terminal; If the total coordination inference latency does not meet the task latency requirements of the local terminal, the resource allocation scheme is adjusted according to the resource allocation feedback information. The resource allocation feedback information includes the total coordination inference latency, transmission latency, resource block utilization, task completion status, and latency constraint satisfaction status of each local terminal.

9. A model inference optimization system that coordinates terminal and edge server, characterized in that, The system includes: The parameter acquisition module is used to acquire the computing power parameters and communication parameters of the local terminal and the edge server in the initial model deployment environment. The computing power parameters are used to characterize the computing performance of the local terminal and the edge server, and the communication parameters are used to characterize the historical communication rate distribution between the local terminal and the edge server. An execution cost calculation module is used to calculate the execution cost of each computing layer in the initial model when it runs on the local terminal or the edge server, by combining the computing power parameters, the communication parameters, and the layer structure information of the initial model; the execution cost includes computing latency and transmission latency; The splitting cost determination module is used to traverse all computing layers in the initial model based on the execution cost of each computing layer, calculate the total inference latency corresponding to the complete inference of the initial model when each computing layer is taken as the model splitting point, and determine the total inference latency as the splitting cost corresponding to the model splitting point. The overall splitting cost calculation module is used to cluster the historical communication rate distribution to obtain multiple central transmission rates, determine the model splitting point corresponding to the minimum splitting cost under each central transmission rate as a candidate splitting point, and obtain the overall splitting cost of the initial model by weighted summation of the splitting costs corresponding to multiple candidate splitting points. The lightweight model generation module is used to optimize the overall splitting cost and model inference accuracy by using reinforcement learning to determine the pruning rate of each computational layer, outputting the optimal pruning strategy that satisfies the accuracy constraint, and processing the initial model based on the optimal pruning strategy to generate a lightweight model. The model deployment module is used to filter out target split points that match the real-time communication rate from the pre-established matching relationship between communication rate ranges and split points. Based on the target split points, the lightweight model is split into terminal-side sub-models and edge-side sub-models. The terminal-side sub-models are deployed on the local terminal, and the edge-side sub-models are deployed on the edge server to carry out collaborative inference.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 8.

Images