A method to accelerate cloud-edge DNN collaborative inference using multiple compression versions

Abstract

This disclosure provides a method for accelerating collaborative inference between cloud-edge DNNs using multiple compressed versions, belonging to the field of computing technology. Specifically, it includes: Step 1, compressing a deep neural network at multiple scales using an existing dataset to obtain multiple DNN models of different sizes. The classification accuracy, output feature map size of each layer, and computational cost of these multiple DNN models are quantified to calculate the total computational cost required locally and at the edge after selecting a segmentation point, thus constructing a quantized model; Step 2, establishing a computational offloading and resource allocation model based on the quantized model; Step 3, optimizing the computational offloading and resource allocation model using a deep reinforcement learning algorithm, balancing latency and accuracy according to the service requirements of the task. This disclosed solution improves inference efficiency, accuracy, adaptability, and security.

Description

Technical Field

[0001] This disclosure relates to the field of computing technology, and in particular to a method for accelerating collaborative inference of cloud-edge-device DNNs with multiple compression versions. Background Technology

[0002] Currently, deep neural networks (DNNs) are widely used in numerous intelligent applications due to their powerful representation capabilities, such as autonomous driving, facial recognition, and information retrieval. Simultaneously, the widespread adoption of the Internet of Things (IoT) has brought a large number of portable intelligent mobile terminals and embedded devices into people's lives. It is conceivable that deploying DNN applications on mobile devices, pushing artificial intelligence capabilities to the network edge, and realizing edge intelligence will be an inevitable trend.

[0003] Edge intelligence deploys artificial intelligence algorithms and models on network edge devices, enabling devices to autonomously process data and make decisions. This significantly reduces data transmission and processing latency, improves application efficiency and real-time performance, and also enhances data security and privacy.

[0004] The fundamental requirement for these intelligent applications is to respond to user queries efficiently and accurately. Therefore, minimizing latency and improving accuracy to ensure a high-quality user experience is crucial. However, DNN-based applications are inherently computationally intensive and resource-intensive. Mobile devices, due to size and cost limitations, are typically equipped with only small computing units. This makes it difficult to execute tasks efficiently and flexibly on resource-constrained mobile devices.

[0005] Traditional cloud computing processes DNN inference tasks by transmitting input data generated by mobile devices to resource-rich remote cloud service centers to meet stringent responsiveness requirements. However, due to unreliable WAN connections and significant latency between mobile devices and the remote cloud, Quality of Service (QoS) remains poor and unsatisfactory. Worse still, for many intelligent applications, their data may contain sensitive and private information, raising privacy and security concerns when transmitting this data to cloud data centers. Edge computing, as a complement to cloud computing, aims to deploy high-performance servers at the network edge, close to mobile devices. Edge computing undoubtedly enhances the QoS and privacy protection of DNN-based applications. However, as task inference requests increase, the limited resources of edge servers lead to increased bandwidth and computing resource usage, ultimately resulting in a decline in QoS.

[0006] A popular and feasible solution is to introduce collaborative inference in edge computing environments. In edge collaborative inference, a complex deep neural network (DNN) can be divided into two parts: the shallow part is processed locally on the mobile device, while the deep part is offloaded to an edge server to complete the remaining inference tasks. For DNNs, intermediate data may be smaller than the source data. In this case, on the one hand, partitioning the DNN model and offloading intermediate features to the edge reduces data transmission volume. On the other hand, performing shallow partitioning locally effectively utilizes the resources of mobile devices, reducing server resource consumption. Therefore, edge intelligence enjoys real-time response and privacy protection through collaborative inference.

[0007] However, this model still presents several key challenges that need to be addressed: (1) how to determine the deployment of DNN partitions in complex heterogeneous environments, (2) how to balance task latency and model accuracy according to different service requirements, and (3) how to reasonably allocate computing and communication resources in collaborative inference on edge devices.

[0008] It is evident that there is an urgent need for a multi-compressed version of cloud-edge-device DNN collaborative inference acceleration method with high inference efficiency, adaptability, and security. Summary of the Invention

[0009] In view of this, the present disclosure provides a multi-compressed version of the cloud-edge-device DNN collaborative inference acceleration method, which at least partially solves the problems of poor inference efficiency, adaptability and security in the prior art.

[0010] This disclosure provides a method for accelerating cloud-edge-device DNN collaborative inference using multiple compression versions, including:

[0011] Step 1: Compress the deep neural network at multiple scales using the existing dataset to obtain multiple versions of the DNN model of different sizes. Quantize the classification accuracy, output feature map size of each layer, and computational cost of the multiple versions of the DNN model to calculate the total computational cost required at the local and edge ends after selecting the segmentation point, and construct the quantized model.

[0012] Step 2: Establish a computational offloading and resource allocation model based on the quantitative model;

[0013] Step 3: Optimize the computation offloading and resource allocation model using deep reinforcement learning algorithms, and balance latency and accuracy according to the service requirements of the task.

[0014] According to a specific implementation of an embodiment of this disclosure, step 1 specifically includes:

[0015] Step 1.1: On a cloud server with abundant computing resources, a collaborative compression method combining channel pruning and tensor decomposition is used to compress the deep neural network at multiple scales by simultaneously learning model sparsity and low rank, resulting in multiple DNN model versions of different sizes. This indicates that m is the number of compression models;

[0016] Step 1.2: Based on the characteristics of the DNN model, the interval between any two adjacent convolutional layers is used as an optional partitioning point;

[0017] Step 1.3: Calculate the accuracy of each version of the DNN model in the public dataset;

[0018] Step 1.4: Calculate the size D of the output feature map of each layer according to the DNN model structure. j,k ,in Indicates the DNN model version. Represents the model partition points;

[0019] Step 1.5: Based on the internal structure and number of fully connected layers, convolutional layers, pooling layers, ReLU layers, and softmax layers in the DNN model, obtain the computational cost of each layer.

[0020] Step 1.6, based on the computational cost of each layer Calculate the local computation cost of the j-th DNN model version at the k-th partition. and edge computing overhead And based on this, a quantitative model is constructed.

[0021] According to a specific implementation of an embodiment of this disclosure, step 2 specifically includes:

[0022] Step 2.1: When the mobile device receives the inference task, it performs model selection and model partitioning for the inference task to form a joint decision result;

[0023] Step 2.2: Perform local inference calculations based on the joint decision results to obtain the local execution delay;

[0024] Step 2.3: Transmit the intermediate feature map data to the edge server at the segmentation point and calculate the network transmission delay;

[0025] Step 2.4: When the edge server receives the intermediate data uploaded by the i-th mobile device, it uses the allocated computing resources to complete the remaining inference of the task, based on the total computing resources F of the edge server and the resource ratio f allocated to the i-th mobile device. i And the expected computational overhead of edge servers performing inference tasks. Calculate the latency of edge inference tasks;

[0026] Step 2.5: Calculate the total latency based on local execution latency, network transmission latency, and edge inference task latency;

[0027] Step 2.6: Calculate the model accuracy for each mobile device in the inference task based on the model compression ratio;

[0028] Step 2.7: Establish a computational offloading and resource allocation model based on total latency and model accuracy.

[0029] According to a specific implementation of this disclosure, the expression for the computational unloading and resource allocation model is:

[0030]

[0031] subject to

[0032] C1:b i ≥0

[0033] C2:

[0034] C3:

[0035] C4:f i ≥0

[0036] C5:

[0037] C6:

[0038] C7:

[0039] C8:

[0040] in, These represent the decisions made by all mobile devices regarding DNN version selection, partition point selection, bandwidth resource allocation, and computing resource allocation, respectively. i T represents the model accuracy. i Let ω1 and ω2 represent the total latency, and let ω1 and ω2 be the coefficients that weight latency and accuracy according to different service requirements. ω1, ω2 ∈ [0, 1], ω1 + ω2 = 1. Constraints C1, C2 and C3 indicate that all bandwidth resources are allocated to mobile devices for data transmission. Constraints C4, C5 and C6 indicate that all computing resources are allocated to mobile devices for computing. Constraint C7 indicates that the compression ratio is selected from the available compression ratios. Constraint C8 indicates that the partition points do not exceed the maximum number of layers in the DNN.

[0041] According to a specific implementation of an embodiment of this disclosure, step 3 specifically includes:

[0042] Step 3.1, define the state space, wherein the state space includes system variables and optimization variables;

[0043] Step 3.2, define the action space, wherein the action space includes unloading decisions and resource allocation decisions, the unloading decisions include compression ratio decisions and split point decisions, and the resource allocation decisions include bandwidth resource allocation decisions and server computing resource decisions;

[0044] Step 3.3, define the reward function;

[0045] Step 3.4: Use deep reinforcement learning strategies to solve the computation offloading and resource allocation model, and generate the optimal strategy to balance latency and accuracy.

[0046] According to a specific implementation of an embodiment of this disclosure, step 3.2 specifically includes:

[0047] Define the compression ratio decision vector This indicates the DNN model version selected by the i-th mobile device;

[0048] Define the split point decision vector This represents the segmentation point selected by the i-th mobile device;

[0049] Define the bandwidth resource allocation decision vector This represents the percentage of bandwidth resources that the i-th mobile device can be allocated.

[0050] Define the computational resource allocation decision vector This represents the percentage of computing resources that the i-th mobile device can be allocated.

[0051] According to a specific implementation of an embodiment of this disclosure, step 3.4 specifically includes:

[0052] Step 3.4.1: In the sampling phase, the near-end policy optimization algorithm is used to collect trajectories through the old policy network in the environment during the sampling phase. The state-action pairs in the trajectory are submitted to the environment to obtain the corresponding reward signal and the next state will be stored in the replay buffer.

[0053] Step 3.4.2: In the optimization phase, the near-end policy optimization algorithm uses the data stored in the replay buffer to train the policy network and the value function network. After the replay buffer is filled, the network is updated K times. For the k-th update, a set of data is extracted from the replay buffer to update the gradient of policy π.

[0054] Step 3.4.3: Repeat steps 3.4.1 and 3.4.2 until the preset conditions are met, and obtain the optimal strategy to balance latency and accuracy.

[0055] According to a specific implementation of an embodiment of this disclosure, step 3.4.2 specifically includes:

[0056] The advantage function is calculated using the generalized advantage estimation method.

[0057] Where, δ t =r t +γV(s t+1 )-V(s t V(s) represents the TD-error at time t. t ) and V(s t+1 ) respectively correspond to state s t and s t+1 The state values ​​are obtained through a value function network, where γ represents the discount factor and λ represents the hyperparameters of the variance and bias of the balancing advantage function.

[0058] Calculate the advantage estimate based on the advantage function.

[0059] in, Let τ represent the probability of the trajectory τ.

[0060] The samples generated by the current policy are calculated based on the advantage estimate. The policy gradient is calculated using a preset shearing policy, and the parameters of the policy network are updated. The gradient of the updated policy π is expressed as:

[0061] Where, p θ and p θ′ Let these represent the distribution functions of the target and the old strategy, respectively. It represents the ratio of the old and new probability distributions.

[0062] According to a specific implementation of this disclosure, before step 3.4.2, the method further includes:

[0063] Using a pre-defined loss function, estimate the value V(s) based on the value function of the current state. t ) and decaying reward V t target The average squared error between the values ​​updates the parameters of the value function network, where the preset loss function is expressed as follows:

[0064]

[0065] in,

[0066] The multi-compressed cloud-edge-device DNN collaborative inference acceleration scheme in this embodiment includes: Step 1, compressing the deep neural network at multiple scales using an existing dataset to obtain multiple DNN models of different sizes, quantizing the classification accuracy, output feature map size of each layer, and computational cost of the multiple DNN models, thereby calculating the total computational cost required locally and at the edge after selecting the segmentation point, and constructing a quantized model; Step 2, establishing a computation offloading and resource allocation model based on the quantized model; Step 3, optimizing the computation offloading and resource allocation model using a deep reinforcement learning algorithm, and balancing latency and accuracy according to the service requirements of the task.

[0067] The beneficial effects of this disclosure are as follows: By compressing the original DNN model at different scales, multiple versions of the DNN model are designed. This method reduces both the computational load and the size of intermediate feature maps, simultaneously optimizing communication and computational latency. Taking into full account the diverse service requirements of the tasks, and aiming to balance latency and accuracy, a multi-dimensional optimization problem model is established. DNN collaborative inference is performed by jointly optimizing model version selection, model partitioning, and resource allocation. Considering the extremely large solution space and NP-hard nature of this optimization problem, a deep reinforcement learning-based algorithm is proposed to solve it. This algorithm can interact with complex dynamic environments, making end-to-end intelligent decisions regarding model version selection, model partitioning, and resource allocation, thereby improving inference efficiency, adaptability, and security. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 A flowchart illustrating a multi-compressed version of a cloud-edge-device DNN collaborative inference acceleration method provided in this embodiment of the disclosure;

[0070] Figure 2 A schematic diagram of a system model for a multi-compressed version of a cloud-edge-device DNN collaborative inference acceleration method provided in this disclosure embodiment;

[0071] Figure 3 A partial flowchart illustrating a multi-compressed version of a cloud-edge-device DNN collaborative inference acceleration method provided in this embodiment of the disclosure;

[0072] Figure 4A partial flowchart illustrating another multi-compression version of the cloud-edge-device DNN collaborative inference acceleration method provided in this embodiment of the disclosure;

[0073] Figure 5 A training performance result diagram provided in an embodiment of this disclosure;

[0074] Figure 6 This is a comparison chart of different latency and accuracy weights provided in an embodiment of the present disclosure. Detailed Implementation

[0075] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0076] The following specific examples illustrate the implementation of this disclosure. Those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0077] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0078] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The drawings only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0079] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0080] This disclosure provides a multi-compression version of a cloud-edge-device DNN collaborative inference acceleration method, which can be applied to data processing in IoT scenarios.

[0081] See Figure 1 This is a flowchart illustrating a multi-compressed version of a cloud-edge-device DNN collaborative inference acceleration method provided in an embodiment of this disclosure. Figure 1 As shown, the method mainly includes the following steps:

[0082] Step 1: Compress the deep neural network at multiple scales using the existing dataset to obtain multiple versions of the DNN model of different sizes. Quantize the classification accuracy, output feature map size of each layer, and computational cost of the multiple versions of the DNN model to calculate the total computational cost required at the local and edge ends after selecting the segmentation point, and construct the quantized model.

[0083] Furthermore, step 1 specifically includes:

[0084] Step 1.1: On a cloud server with abundant computing resources, a collaborative compression method combining channel pruning and tensor decomposition is used to compress the deep neural network at multiple scales by simultaneously learning model sparsity and low rank, resulting in multiple DNN model versions of different sizes. This indicates that m is the number of compression models;

[0085] Step 1.2: Based on the characteristics of the DNN model, the interval between any two adjacent convolutional layers is used as an optional partitioning point;

[0086] Step 1.3: Calculate the accuracy of each version of the DNN model in the public dataset;

[0087] Step 1.4: Calculate the size D of the output feature map of each layer according to the DNN model structure. j,k ,in Indicates the DNN model version. Represents the model partition points;

[0088] Step 1.5: Based on the internal structure and number of fully connected layers, convolutional layers, pooling layers, ReLU layers, and softmax layers in the DNN model, obtain the computational cost of each layer.

[0089] Step 1.6, based on the computational cost of each layer Calculate the local computation cost of the j-th DNN model version at the k-th partition. and edge computing overhead And based on this, a quantitative model is constructed.

[0090] This disclosure considers a multi-user, multi-task DNN inference acceleration scenario, such as... Figure 2 As shown, considering the diversity of task service requirements, a multi-dimensional joint optimization problem model is established, including model version, model partitioning, and resource allocation. A cloud-edge collaborative DNN inference acceleration framework is proposed. In this method, the DNN adaptively selects its version and offloads partitions, and utilizes the computational power of MD (Multi-Dimensional Model) to allocate resources at the edge for collaborative inference, thereby accelerating DNN inference and balancing latency and accuracy according to the task's service requirements.

[0091] In practice, constructing a quantitative model may include the following steps:

[0092] Step 101: On a cloud server with abundant computing resources, a collaborative compression method combining channel pruning and tensor decomposition is used. By simultaneously learning the sparsity and low rank of the model, the deep neural network is compressed at multiple scales, resulting in multiple DNN model versions of different sizes. This indicates that m is the number of compressed models. These models will be deployed simultaneously on mobile devices and edge servers for subsequent edge-to-edge collaborative inference.

[0093] Step 102: Based on the characteristics of the DNN model, we use the interval between any two adjacent convolutional layers as optional partitioning points. It's worth noting that some layers with non-cascaded connections, such as residual blocks, are integrated into a single layer unit. The partitioning point divides the model into two parts: the part before the partitioning point is executed on the mobile device, and the intermediate feature map output from the mobile device is then transmitted wirelessly to the edge, where the remaining inference is completed. Optional Partitioning Point Set Where n is the number of optional partition points. When the number of partition points is 1, it means that the entire inference task will be offloaded to the edge server to be completed; while when the number of partition points is n, it means that the entire inference task will be performed locally.

[0094] Step 103: For each version of the DNN model, calculate the corresponding accuracy in the public dataset, denoted as... These are used to evaluate the inference accuracy performance of these DNN models.

[0095] Step 104: Calculate the size D of the output feature map of each layer based on the model structure. j,k ,in Indicates the DNN model version. This represents the model partition point.

[0096] Step 105: The computational cost of a DNN can be estimated by considering the number of FLOPs required for each neural network layer. By analyzing the internal structure and number of fully connected layers, convolutional layers, pooling layers, ReLU layers, and softmax layers in the DNN model, the computational cost f of each layer can be obtained. l j .

[0097] Step 106: Calculate the cost f for each layer based on the calculations in Step 105. l j Calculate the local computation cost of the j-th DNN model version at the k-th partition. and edge computing overhead

[0098] Step 2: Establish a computational offloading and resource allocation model based on the quantitative model;

[0099] Based on the above embodiments, step 2 specifically includes:

[0100] Step 2.1: When the mobile device receives the inference task, it performs model selection and model partitioning for the inference task to form a joint decision result;

[0101] Step 2.2: Perform local inference calculations based on the joint decision results to obtain the local execution delay;

[0102] Step 2.3: Transmit the intermediate feature map data to the edge server at the segmentation point and calculate the network transmission delay;

[0103] Step 2.4: When the edge server receives the intermediate data uploaded by the i-th mobile device, it uses the allocated computing resources to complete the remaining inference of the task, based on the total computing resources F of the edge server and the resource ratio f allocated to the i-th mobile device. i And the expected computational overhead of edge servers performing inference tasks. Calculate the latency of edge inference tasks;

[0104] Step 2.5: Calculate the total latency based on local execution latency, network transmission latency, and edge inference task latency;

[0105] Step 2.6: Calculate the model accuracy for each mobile device in the inference task based on the model compression ratio;

[0106] Step 2.7: Establish a computational offloading and resource allocation model based on total latency and model accuracy.

[0107] Furthermore, the expression for the computational unloading and resource allocation model is as follows:

[0108]

[0109] subject to

[0110] C1:b i ≥0

[0111] C2:

[0112] C3:

[0113] C4:f i ≥0

[0114] C5:

[0115] C6:

[0116] C7:

[0117] C8:

[0118] in, These represent the decisions made by all mobile devices regarding DNN version selection, partition point selection, bandwidth resource allocation, and computing resource allocation, respectively. i T represents the model accuracy. i Let ω1 and ω2 represent the total latency, and let ω1 and ω2 be the coefficients that weight latency and accuracy according to different service requirements. ω1, ω2 ∈ [0, 1], ω1 + ω2 = 1. Constraints C1, C2 and C3 indicate that all bandwidth resources are allocated to mobile devices for data transmission. Constraints C4, C5 and C6 indicate that all computing resources are allocated to mobile devices for computing. Constraint C7 indicates that the compression ratio is selected from the available compression ratios. Constraint C8 indicates that the partition points do not exceed the maximum number of layers in the DNN.

[0119] In specific implementation, such as Figure 3 As shown, the steps for establishing a computational offloading and resource allocation model based on the quantization model may include the following:

[0120] Step 201, as follows Figure 3As shown in Figure ①, when a mobile device receives a task, it needs to perform model selection and model partitioning for the inference task. In the model selection step, a smaller model can be chosen for inference to reduce latency, but this inevitably leads to a loss of accuracy. In the model partitioning stage, a selected compressed model is divided into two parts: the part before the split is deployed locally, and the part after the split is deployed to the edge server. The selected compressed model and the offloading partition ratio may vary significantly between heterogeneous terminal devices. Therefore, the edge server needs to allocate computing resources (… Figure 3 As shown in ②), bandwidth resources ( Figure 3 As shown in ③, adaptive allocation is performed. Figure 3 As shown in diagrams ④, ⑤, and ⑥, after obtaining the joint decision result, collaborative inference will begin from the mobile device and transmit intermediate data to the edge server at the split point to complete the subsequent inference. After the inference is completed, the edge server will return the result to the mobile device.

[0121] Step 202, Local Inference Computation. After the mobile device receives the task, it will first perform inference computation locally. The local inference time cost generated by the inference task of the i-th mobile device is determined by its local computing power F. i Expected computational overhead for locally executed inference tasks Decision. Therefore, the local execution delay T of the i-th MD. i l It can be represented as:

[0122] Step 203, intermediate feature map transmission. If the task partition point is not n, i.e., the task is not executed entirely locally, the intermediate feature map generated by the network layer before the partition point will be transmitted to the edge server to complete the subsequent inference task. For communication, since the tasks on each device have the same priority, the tasks will be unloaded simultaneously. We ignore queuing latency. The mobile devices and the edge server are connected via a wireless network. When a task arrives, we will adaptively allocate network resources to each device in a dynamic environment. The transmission rate of the i-th mobile device is:

[0123] Where σ represents the background noise power, b i U represents the percentage of bandwidth resources allocated to the i-th mobile device, B represents the total bandwidth resources in the channel, and u represents the percentage of bandwidth resources allocated to the i-th mobile device. i h represents the transmission power of the data uploaded by the i-th mobile device. i This represents the wireless channel gain between the i-th mobile device and the edge server ES. Its value is related to the communication distance between the i-th mobile device and the edge server ES. The channel gain between the i-th mobile device and ES is expressed as follows: disti This represents the communication distance between the i-th mobile device and the edge server ES. This represents the path loss index.

[0124] Therefore, based on the size of the intermediate feature map data And transmission rate r, network transmission delay T i c It can represent

[0125] as follows:

[0126] Step 204, Edge Inference Computation. When the ES receives the intermediate data uploaded by the i-th MD, it uses the allocated computing resources to complete the remaining inference of the task. Based on the total computing resources F of the ES, the resource ratio f allocated to the i-th MD is determined. i And the expected computational overhead of ES performing inference tasks. Delay T of edge reasoning task i e It can be defined as follows:

[0127] Step 205, Calculate the total latency. It is obvious that the total time cost T for the inference of the i-th MD is... i for:

[0128] T i =T i l +T i c +T i e

[0129] Step 206, Model Accuracy Calculation. In the inference task, the model accuracy Acc of the i-th MD is calculated. i Only with

[0130] The compression ratio of the model is relevant, and the result can be expressed as:

[0131] Step 207, Problem Formulation. To improve QoS, an effective computation offloading and resource allocation strategy must be designed. The goal of our multidimensional optimization problem is to find an optimal balance between accuracy and latency, based on the service requirements of the task, considering concurrent requests from all mobile devices, to maximize the overall performance of the system. To achieve this goal, we consider several factors, such as the selection of model version, the selection of model partitioning points, and the allocation of bandwidth and computational resources. The specific optimization objective is expressed as:

[0132]

[0133] subject to

[0134] C1:b i ≥0

[0135] C2:

[0136] C3:

[0137] C4:f i ≥0

[0138] C5:

[0139] C6:

[0140] C7:

[0141] C8:

[0142] in These represent the decisions made by all MDs regarding DNN version selection, partition selection, bandwidth resources, and computational resource allocation. ω1 and ω2 are weighted coefficients for latency and accuracy based on different service requirements, ω1,ω2∈[0,1], ω1+ω2=1. Constraints C1, C2, and C3 indicate that all bandwidth resources are allocated to MDs for data transmission, while constraints C4, C5, and C6 indicate that all computational resources are allocated to MDs for computation. Constraint C7 indicates that the compression ratio is selected from the available compression ratios, and constraint C8 indicates that the partition number does not exceed the maximum number of layers in the DNN.

[0143] Clearly, P1 is a non-convex mixed-integer programming problem with discrete variables. and and continuous variables and Our goal is to develop an optimization strategy that maximizes system performance while ensuring high accuracy and low latency required for task service. To achieve this, we propose a global end-to-end optimization method based on deep reinforcement learning.

[0144] Step 3: Optimize the computation offloading and resource allocation model using deep reinforcement learning algorithms, and balance latency and accuracy according to the service requirements of the task.

[0145] Based on the above embodiments, step 3 specifically includes:

[0146] Step 3.1, define the state space, wherein the state space includes system variables and optimization variables;

[0147] Step 3.2, define the action space, wherein the action space includes unloading decisions and resource allocation decisions, the unloading decisions include compression ratio decisions and split point decisions, and the resource allocation decisions include bandwidth resource allocation decisions and server computing resource decisions;

[0148] Step 3.3, define the reward function;

[0149] Step 3.4: Use deep reinforcement learning strategies to solve the computation offloading and resource allocation model, and generate the optimal strategy to balance latency and accuracy.

[0150] Furthermore, step 3.2 specifically includes:

[0151] Define the compression ratio decision vector This indicates the DNN model version selected by the i-th mobile device;

[0152] Define the split point decision vector This represents the segmentation point selected by the i-th mobile device;

[0153] Define the bandwidth resource allocation decision vector This represents the percentage of bandwidth resources that the i-th mobile device can be allocated.

[0154] Define the computational resource allocation decision vector This represents the percentage of computing resources that the i-th mobile device can be allocated.

[0155] Furthermore, step 3.4 specifically includes:

[0156] Step 3.4.1: In the sampling phase, the near-end policy optimization algorithm is used to collect trajectories through the old policy network in the environment during the sampling phase. The state-action pairs in the trajectory are submitted to the environment to obtain the corresponding reward signal and the next state will be stored in the replay buffer.

[0157] Step 3.4.2: In the optimization phase, the near-end policy optimization algorithm uses the data stored in the replay buffer to train the policy network and the value function network. After the replay buffer is filled, the network is updated K times. For the k-th update, a set of data is extracted from the replay buffer to update the gradient of policy π.

[0158] Step 3.4.3: Repeat steps 3.4.1 and 3.4.2 until the preset conditions are met, and obtain the optimal strategy to balance latency and accuracy.

[0159] Furthermore, step 3.4.2 specifically includes:

[0160] The advantage function is calculated using the generalized advantage estimation method.

[0161] Where, δ t =r t +γV(s t+1 )-V(s t V(s) represents the TD-error at time t. t ) and V(s t+1 ) respectively correspond to state s t and s t+1 The state values ​​are obtained through a value function network, where γ represents the discount factor and λ represents the hyperparameters of the variance and bias of the balancing advantage function.

[0162] Calculate the advantage estimate based on the advantage function.

[0163] in, Let τ represent the probability of the trajectory τ.

[0164] The samples generated by the current policy are calculated based on the advantage estimate. The policy gradient is calculated using a preset shearing policy, and the parameters of the policy network are updated. The gradient of the updated policy π is expressed as:

[0165] Where, p θ and p θ′ Let these represent the distribution functions of the target and the old strategy, respectively. It represents the ratio of the old and new probability distributions.

[0166] Furthermore, prior to step 3.4.2, the method further includes:

[0167] Using a pre-defined loss function, estimate the value V(s) based on the value function of the current state. t ) and decaying reward V t target The average squared error between the values ​​updates the parameters of the value function network, where the preset loss function is expressed as follows:

[0168]

[0169] in,

[0170] In practice, the optimization process for the computational offloading and resource allocation model can be shown in the following steps:

[0171] Step 301, define the state space.

[0172] In the context of DRL, the state space needs to contain sufficient information to achieve intelligent decision-making. Therefore, we divide the state into two parts: system variables and optimization variables. The former includes relevant parameter settings in our system environment, including... The latter includes total system delay and total accuracy, denoted by {T}. i Acc i} represents the delay T. i Accuracy i The guided algorithm strikes a balance between these two aspects and achieves the optimal goal. Therefore, we define the state as:

[0173] Step 302: Define the action space.

[0174] The action space is divided into two parts: offloading decisions and resource allocation decisions. Offloading decisions include compression ratio decisions and split point decisions. Resource allocation decisions include bandwidth resource allocation decisions and server computing resource decisions. We define a compression ratio decision vector. Indicates the DNN model version selected by the i-th mobile device; the split point decision vector. Represents the split point selected by the i-th mobile device; bandwidth resource allocation decision vector. This represents the percentage of bandwidth resources that the i-th mobile device can be allocated; calculate the resource allocation decision vector. This represents the proportion of computing resources that the i-th mobile device can be allocated. In the t-th iteration, the specific action decision can be expressed as:

[0175]

[0176]

[0177]

[0178]

[0179] Step 303: Define the reward function.

[0180] The reward function plays a crucial role in reinforcement learning. It's the signal an agent receives from the environment after performing an action, used to evaluate the correctness of the agent's current behavior. The goal is for the agent to learn to maximize the long-term accumulated reward value at each time step. Considering the optimization objective of our task, we should use the reward function to guide the policy π towards the optimization objective of increasing accuracy and reducing latency. Let r... t Let t be the reward that the agent can obtain at time t, which can be expressed as:

[0181]

[0182] ω1 and ω2 are coefficients used to weight latency and accuracy based on different service requirements. and These are the standardized values ​​for total system delay and accuracy, respectively.

[0183] Step 304: Solve using deep reinforcement learning strategies.

[0184] Deep reinforcement learning is a method that combines deep learning and reinforcement learning. Its core idea is to enable an agent to interact with its environment and use a deep neural network as a function approximator, allowing the agent to autonomously explore, learn, and master the optimal policy in an unknown environment. * Proximal Policy Optimization (PPO) is the deep reinforcement learning prototype used in our method. It avoids the tedious process of manually setting features, can handle high-dimensional continuous state-action spaces and complex decision problems, and has high sample utilization through importance sampling.

[0185] PPO is a policy-based DRL algorithm with a critic-actor structure, consisting of two policy networks—new and old policies—and a value function network. The new policy network π... θ Characterized by a set of network parameters θ, while the old policy network π θ′ Characterized by the group network parameters θ′, it has the same structure as the new policy network, updating differences through multiple iterations, and is initialized and updated based on θ. Value function network π φ Characterized by a set of network parameters φ. The policy network outputs the probability distribution of the actions the intelligent agent may take in a specific state, while the value function network estimates the value of the current state by calculating the expected long-term reward the agent will receive in that state. The specific algorithm flow is as follows: Figure 4 As shown.

[0186] The PPO algorithm has two main phases: the sampling phase and the optimization phase. In the sampling phase, the algorithm collects a series of trajectories through the old policy network in the environment, denoted as τ={s1,a1,s2,s2,...,s...}. T ,a T Then, it submits the state-action pairs in these trajectories to the environment to obtain the corresponding reward signal and the next state, and sends this data... t ,a t ,r t The data is stored in the replay buffer. During the optimization phase, the PPO algorithm uses the data stored in the replay buffer to train the policy network and the value function network. After the replay buffer is full, the network is updated K times. For the k-th update, a set of data is drawn from the replay buffer to update the gradient of π. To compute the advantage function, GAE (Generalized Advantage Estimation) is used:

[0187]

[0188] Where δ t =r​t +γV(s t+1 )-V(s t V(s) represents the TD-error at time t. t ) and V(s t+1 ) respectively correspond to state s t and s t+1 The state values ​​are obtained through a value function network. The discount factor γ determines the importance of future rewards. Furthermore, λ is a hyperparameter balancing the variance and bias of the advantage function. The GAE function is calculated by assigning decreasing weights to the advantage function over time steps. This makes the importance of the advantage estimate to future rewards gradually decrease. This can be mathematically expressed as: in This represents the probability of the trajectory τ.

[0189] After calculating the advantage estimates, the PPO algorithm uses them to compute the policy gradient and update the parameters of the policy network. Here, p θ and p θ′ Let represent the distribution functions of the target and the old policy, respectively. The gradient used for updating can be expressed as:

[0190] in This represents the ratio of the old to the new probability distribution. In the initial stage of training, R... t =1, because θ and θ′ are consistent and have the same distribution.

[0191] To update the policy, we need samples generated by the current policy. However, updating the policy while using these samples leads to inconsistencies between the current and updated policies, ultimately affecting the effectiveness of the update. To address this issue, the PPO algorithm employs a pruning strategy. The network loss is calculated as follows:

[0192]

[0193] Here, ∈ is a hyperparameter that controls the extent to which the new policy can deviate from the old policy. The clip(.) function used in the PPO algorithm limits the magnitude of change of the new policy relative to the old policy. Clipping ensures that the new policy does not deviate significantly from the old policy and ensures that updates are performed in a stable manner.

[0194] Furthermore, to improve the accuracy of state value prediction during policy network updates, the PPO algorithm trains a value function network. During the training of the value function network, the value function estimate V(s) of the current state is used. t ) and decaying reward V t targetThe mean squared error between the two values ​​updates the network parameters. This helps the network better predict the cumulative value of future rewards, which in turn helps the policy network choose actions that maximize long-term rewards. The loss function is calculated as follows:

[0195]

[0196]

[0197] As the model iterates through sampling and optimization phases, the algorithm converges, and the optimal strategy π is reached. * To be learned.

[0198] like Figure 5 As shown in the simulation experiments conducted on heterogeneous devices, the proposed scheme achieves a better balance between latency and accuracy based on the service requirements of the task as the number of iterations increases. Furthermore, when setting different latency and accuracy weights, the results of the generated optimal strategy balancing latency and accuracy are as follows: Figure 6 As shown.

[0199] Simulation experiments were conducted on heterogeneous devices and compared with various baseline methods. The results, shown in Table 1, demonstrate the superior performance of the proposed method.

[0200]

[0201] Table 1

[0202] This embodiment provides a multi-compressed version of the cloud-edge-device DNN collaborative inference acceleration method. By compressing the original DNN model at different scales, multiple versions of the DNN model are designed. This method reduces both the computational load and the size of intermediate feature maps, simultaneously optimizing communication and computational latency. Taking into full account the diverse service requirements of the tasks, and aiming to balance latency and accuracy, a multi-dimensional optimization problem model is established. DNN collaborative inference is performed by jointly optimizing model version selection, model partitioning, and resource allocation. Considering the extremely large solution space and NP-hard nature of this optimization problem, a deep reinforcement learning-based algorithm is proposed to solve it. This algorithm can interact with complex dynamic environments, making end-to-end intelligent decisions regarding model version selection, model partitioning, and resource allocation, thus improving inference efficiency, adaptability, and security.

[0203] The units described in the embodiments of this disclosure can be implemented in software or in hardware.

[0204] It should be understood that the various parts of this disclosure can be implemented in hardware, software, firmware, or a combination thereof.

[0205] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for accelerating collaborative inference between cloud-edge-device DNNs with multiple compression versions, characterized in that, include: Step 1: Compress the deep neural network at multiple scales using the existing dataset to obtain multiple versions of the DNN model of different sizes. Quantize the classification accuracy, output feature map size of each layer, and computational cost of the multiple versions of the DNN model to calculate the total computational cost required at the local and edge ends after selecting the segmentation point, and construct the quantized model. Step 2: Establish a computational offloading and resource allocation model based on the quantitative model; Step 2 specifically includes: Step 2.1: When the mobile device receives the inference task, it performs model selection and model partitioning for the inference task to form a joint decision result; Step 2.2: Perform local inference calculations based on the joint decision results to obtain the local execution delay; Step 2.3: Transmit the intermediate feature map data to the edge server at the segmentation point and calculate the network transmission delay; Step 2.4: When the edge server receives the intermediate data uploaded by the i-th mobile device, it uses the allocated computing resources to complete the remaining inference of the task, based on the total computing resources F of the edge server and the resource ratio f allocated to the i-th mobile device. i And the expected computational overhead of edge servers performing inference tasks. Calculate the latency of edge inference tasks; Step 2.5: Calculate the total latency based on local execution latency, network transmission latency, and edge inference task latency; Step 2.6: Calculate the model accuracy for each mobile device in the inference task based on the model compression ratio; Step 2.7: Establish a computation offloading and resource allocation model based on total latency and model accuracy; Step 3: Optimize the computation offloading and resource allocation model using deep reinforcement learning algorithms, and balance latency and accuracy according to the service requirements of the task.

2. The method according to claim 1, characterized in that... Step 1 specifically includes: Step 1.1: On a cloud server with abundant computing resources, a collaborative compression method combining channel pruning and tensor decomposition is used to compress the deep neural network at multiple scales by simultaneously learning model sparsity and low rank, resulting in multiple DNN model versions of different sizes. This indicates that m is the number of compression models; Step 1.2: Based on the characteristics of the DNN model, the interval between any two adjacent convolutional layers is used as an optional partitioning point; Step 1.3: Calculate the accuracy of each version of the DNN model in the public dataset; Step 1.4: Calculate the size D of the output feature map of each layer according to the DNN model structure. j,k ,in Indicates the DNN model version. Represents the model partition points; Step 1.5: Based on the internal structure and number of fully connected layers, convolutional layers, pooling layers, ReLU layers, and softmax layers in the DNN model, obtain the computational cost of each layer. Step 1.6, based on the computational cost of each layer Calculate the local computation cost of the j-th DNN model version at the k-th partition. and edge computing overhead And based on this, a quantitative model is constructed.

3. The method according to claim 2, characterized in that... The expression for the computational unloading and resource allocation model is as follows: in, These represent the decisions made by all mobile devices regarding DNN version selection, partition point selection, bandwidth resource allocation, and computing resource allocation, respectively. i T represents the model accuracy. i Let ω1 and ω2 represent the total latency, and let ω1 and ω2 be the coefficients that weight latency and accuracy according to different service requirements. ω1, ω2 ∈ [0, 1], ω1 + ω2 = 1. Constraints C1, C2 and C3 indicate that all bandwidth resources are allocated to mobile devices for data transmission. Constraints C4, C5 and C6 indicate that all computing resources are allocated to mobile devices for computing. Constraint C7 indicates that the compression ratio is selected from the available compression ratios. Constraint C8 indicates that the partition points do not exceed the maximum number of layers in the DNN.

4. The method according to claim 3, characterized in that... Step 3 specifically includes: Step 3.1, define the state space, wherein the state space includes system variables and optimization variables; Step 3.2, define the action space, wherein the action space includes unloading decisions and resource allocation decisions, the unloading decisions include compression ratio decisions and split point decisions, and the resource allocation decisions include bandwidth resource allocation decisions and server computing resource decisions; Step 3.3, define the reward function; Step 3.4: Use deep reinforcement learning strategies to solve the computation offloading and resource allocation model, and generate the optimal strategy to balance latency and accuracy.

5. The method according to claim 4, characterized in that... Step 3.2 specifically includes: Define the compression ratio decision vector This indicates the DNN model version selected by the i-th mobile device; Define the split point decision vector This represents the segmentation point selected by the i-th mobile device; Define the bandwidth resource allocation decision vector This represents the percentage of bandwidth resources that the i-th mobile device can be allocated. Define the computational resource allocation decision vector This represents the percentage of computing resources that the i-th mobile device can be allocated.

6. The method according to claim 5, characterized in that... Step 3.4 specifically includes: Step 3.4.1: In the sampling phase, the near-end policy optimization algorithm is used to collect trajectories through the old policy network in the environment during the sampling phase. The state-action pairs in the trajectory are submitted to the environment to obtain the corresponding reward signal and the next state, which are then stored in the replay buffer. Step 3.4.2: In the optimization phase, the near-end policy optimization algorithm uses the data stored in the replay buffer to train the policy network and the value function network. After the replay buffer is filled, the network is updated K times. For the k-th update, a set of data is extracted from the replay buffer to update the gradient of policy π. Step 3.4.3: Repeat steps 3.4.1 and 3.4.2 until the preset conditions are met, and obtain the optimal strategy to balance latency and accuracy.

7. The method according to claim 6, characterized in that... Step 3.4.2 specifically includes: The advantage function is calculated using the generalized advantage estimation method. Where, δ t =r t +γV(s t+1 )-V(s t V(s) represents the TD-error at time t. t ) and V(s t+1 ) respectively correspond to state s t and s t+1 The state values ​​are obtained through a value function network, where γ represents the discount factor and λ represents the hyperparameters of the variance and bias of the balancing advantage function. Calculate the advantage estimate based on the advantage function. in, Represents the probability of trajectory τ; The samples generated by the current policy are calculated based on the advantage estimate. The policy gradient is calculated using a preset shearing policy, and the parameters of the policy network are updated. The gradient of the updated policy π is expressed as: Where, p θ and p θ′ Let represent the distribution functions of the target and the old strategy, respectively. It represents the ratio of the old and new probability distributions.

8. The method according to claim 7, characterized in that Before step 3.4.2, the method further includes: Using a pre-defined loss function, estimate the value V(s) based on the value function of the current state. t ) and decaying reward V t target The average squared error between the values ​​updates the parameters of the value function network, where the preset loss function is expressed as follows: in,

Images