A real-time edge-cloud collaborative convolutional neural network inference method and system

Abstract

A real-time edge-cloud collaborative convolutional neural network inference method and system includes the following steps: converting a compressed CNN model into an edge-cloud collaborative inference model SW-CNN; obtaining the communication computation delay of each layer of the SW-CNN at each compression rate; adjusting the compression rate c according to the real-time network performance; modeling and solving the real-time edge-cloud collaborative inference control problem, and selecting a compression rate for the SW-CNN; adjusting the compression rate of the SW-CNN to c n , analysis task p n The present invention divides the CNN model into two parts: the edge and the cloud. It compresses the original data through a small amount of model calculation at the edge, and then uploads the intermediate data of the CNN model output by the edge to the cloud. A large amount of model calculation is completed in the cloud to obtain the final inference result, realizing high-precision and high-responsive real-time CNN inference. It dynamically adjusts the model compression rate according to the real-time task load and network environment, and supports real-time streaming intelligent analysis in a fluctuating network environment.

Description

Technical Field

[0001] The present invention belongs to the field of distributed intelligence, and specifically relates to a real-time edge-cloud collaborative convolutional neural network reasoning method and system. Background Art

[0002] With the advancement of deep learning, convolutional neural network (CNN) inference has achieved remarkable results across a range of tasks, even surpassing human performance in some vision tasks. Accurate and real-time CNN inference supports numerous practical applications, such as pedestrian recognition using surveillance cameras, object tracking using drones, and traffic light recognition using autonomous vehicles, driven by the massive amounts of high-resolution images continuously generated by widely distributed Internet of Things (IoT) devices. However, the real-time efficiency (throughput at high accuracy) of CNN inference in existing approaches remains severely limited within mainstream edge or cloud computing paradigms. Specifically, the efficiency of existing edge computing approaches is constrained by the computationally intensive CNN models and resource-constrained edge devices. The efficiency of existing cloud computing approaches is also constrained by the massive amounts of high-resolution edge images and limited edge-cloud bandwidth. Edge-cloud collaborative inference can effectively improve efficiency, but its efficiency is affected by network fluctuations. A fixed edge-cloud collaborative solution cannot meet the demands of real-time streaming processing in fluctuating network environments. Summary of the Invention

[0003] The purpose of the present invention is to provide a real-time edge-cloud collaborative convolutional neural network inference method and system to solve the problem that the efficiency of existing cloud computing methods is limited by massive high-resolution edge images and limited edge-cloud bandwidth.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A real-time edge-cloud collaborative convolutional neural network inference method includes the following steps:

[0006] On the edge device, the CNN model is compressed by feature map and converted into the edge-cloud collaborative inference model SW-CNN.

[0007] Information collection is performed in a given edge-cloud system to obtain the communication computation delay of each layer of SW-CNN at various compression rates;

[0008] Model the real-time edge-cloud collaborative CNN inference control problem and adjust the compression ratio c based on the real-time network performance for streaming inference tasks arriving in the time sequence i=1,2...;

[0009] Initialize the task queue. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on the current task remaining time information and real-time network conditions. It selects the compression rate for SW-CNN and adjusts the compression rate of SW-CNN to c n , analysis task p n ;

[0010] Task p n After completion, if the task queue is not empty, the compression rate of SW-CNN is repeatedly adjusted to c n , process task p n , until all tasks in the task queue are processed.

[0011] Furthermore, the feature map compression of the CNN model on the edge device specifically includes:

[0012] Given that the communication data between CNN layers is a feature map, the amount of data D s =F s *F n *F b , where F s Represents the area of ​​a single feature map, F n Indicates the number of feature maps, F b Indicates the number of bits of quantized data, respectively reducing F n and F b , assuming that the number of feature maps remaining after the compression operation is F rn , the remaining data quantization bits are F rb , then (F rn ,F rb ) is the current compression setting, the communication compression ratio of this setting is c=1-(F rn *F rb ) / (F n *F b ), the residual rate r = 1-c. After compression, the model is fine-tuned on the original dataset through end-to-end training to restore the accuracy, and the Pareto set of compression rate-accuracy is calculated to determine the solution with the highest accuracy under the same compression rate.

[0013] Furthermore, channel pruning ChannelPruning and uniform affine quantization UniformAffineQuantization are used to reduce F n and F b .

[0014] Furthermore, converting the compressed CNN model into the edge-cloud collaborative inference model SW-CNN specifically includes:

[0015] SW-CNN conversion, for a given CNN model, obtains CNN models with different compression rates and accuracy, converts the given CNN model into a SW-CNN model that integrates CNN models with different compression rates. For a given CNN model, its corresponding SW-CNN model has the same structure. For a given CNN model with F N and F B The partition layer generates different sub-models according to different compression settings. Except for the given partition layer, different sub-models share all parameters. For the compression layer, different sub-models are based on (F rn ,F rb ) are nested and share parameters, using (F rn ,F rb ) The sub-models of the compression setting use numbers from 1 to F rn Feature maps and 1 to F rb The number of data bits, so (F rn ,F rb )The large sub-model contains a sub-model with a small number of feature maps and compression bits, that is, the small sub-model is nested in the large sub-model.

[0016] Furthermore, for a given CNN model, a CNN model with the same structure as its corresponding SW-CNN is first trained, and its parameters are used to initialize the SW-CNN model corresponding to the given CNN model; the pre-trained model is used as a teacher model to guide the training of the remaining sub-models; the SW-CNN is iteratively trained, and in each training iteration, all sub-models independently complete their forward reasoning and backpropagation processes, and then the gradients are summed to update the SW-CNN model. At the beginning of the first epoch of training, the gradients of different sub-models are clustered using DBSCAN to group the sub-models. In subsequent training, only one model is sampled in each group for training and updating during each iterative update. The number of sub-models trained is the number of sub-model groups, which is adjusted by the eps parameter in the DBSCAN algorithm.

[0017] Furthermore, calculate the delay:

[0018] Deploy an N-layer CNN model in a 2-layer edge-cloud system, with the partitioning layer being l and the compression rate being c. CNN models from 0 to l layers run on edge devices, and CNN models from l+1 to N layers run on cloud servers. Under this partitioning and compression scheme, the end-to-end latency of CNN inference is in is the sum of the delays of the CNN model inference on the edge, T t (l)*c is the sum of the communication delays between edge and cloud, It is the sum of the inference delays of the remaining CNN models on the cloud. Different (l, c) correspond to different inference delays and accuracies. l is specified by the user. The given CNN is converted into a SW-CNN. c is selected from the compression rate-accuracy Pareto set of the SW-CNN.

[0019] Furthermore, adjust the compression ratio:

[0020] When the previous task p n-1 When completed, the controller uses the current number of tasks p n , the remaining time left by the previous task for the current task l n-1 and the current network bandwidth b n As input, determine the current compression rate c for the batch of tasks n To maximize the real-time performance index RP, if task p n-1 exist When the time is completed, the next batch of tasks p n Completion time where r(c n ,p n ) represents the computation time of the batch of tasks, Indicates the communication time, taking into account the user-limited delay l user , task p n The remaining time is Where (x) - =min{x,l user} to ensure that the remaining time is not greater than the time given by the user, Δt (n-1,n) is the time interval between the arrival of two batches of tasks, if l n <0Task p n There must be a delay, for tasks p1 to p n , the average precision is The delay rate is in() * represents the step function, and the control objective function is Where γ is a non-negative real number, which indicates the user's preference for real-time control. The larger γ is, the more the control tends to reduce the delay rate and can tolerate higher precision loss. The real-time reasoning control problem is modeled as solving

[0021]

[0022]

[0023] (l0=l user )

[0024] c1,c2,…,c N ∈C Pareto

[0025] Among them, C Pareto It means that only the compression rate in the Pareto set is selected to ensure the highest reasoning accuracy under the same compression rate. l0 represents the remaining time of the first task. If there is no backlog task, it can be processed immediately. l0 is equal to l user ; If there is a backlog task, and the current backlog task has arrived at time t0, then l0=l user -t0.

[0026] Furthermore, the compression rate is selected for SW-CNN

[0027] If the task queue is empty, suspend; otherwise, load the current inference task p from the queue n , sample the current bandwidth b n , the remaining time of the previous task Based on the current queue length N, the model predictive control algorithm is used to predict the tasks and bandwidth at N time points in the future. Where N is equal to the number of backlogged task batches. We model the real-time edge-cloud collaborative reasoning control problem. We transform the planning optimization problem into a combinatorial optimization problem by using discrete bandwidth data. We use the simulated annealing algorithm to search for the optimal solution. When the backlog of tasks increases, we select a higher compression rate to ensure real-time performance. Let the complete compression rate set be C. Pareto When N batches of tasks are backlogged in the queue, Pareto Delete the first N smaller compression rates in ascending order to obtain the current feasible compression rate set C' Pareto , and make the capacity of the feasible set card(C' Pareto ) is equal to (card(C Pareto )-N) + , when N=1, the search space is equal to card(C Pareto ), as N increases, the capacity of the feasible set card (C' Pareto ) gradually increases.

[0028] Furthermore, a real-time edge-cloud collaborative convolutional neural network inference system includes:

[0029] The model conversion module is used to compress the CNN model through feature maps on the edge device and convert the compressed CNN model into the edge-cloud collaborative inference model SW-CNN;

[0030] The delay calculation module is used to collect information in a given edge-cloud system to obtain the communication calculation delay of each layer of the SW-CNN at various compression rates;

[0031] The compression rate adjustment module is used to model the real-time edge-cloud collaborative CNN inference control problem and adjust the compression rate c of streaming inference tasks arriving in the time sequence i=1,2... according to the real-time network performance;

[0032] The compression rate selection module is used to initialize the task queue. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on the current task remaining time information and real-time network conditions, and selects the compression rate for SW-CNN; adjusts the compression rate of SW-CNN to c n , analysis task p n .

[0033] Compared with the prior art, the present invention has the following technical effects:

[0034] The present invention divides the CNN model into two parts, the edge and the cloud. It compresses the original data through a small amount of model calculation at the edge, and then uploads the intermediate data of the CNN model output at the edge to the cloud. A large amount of model calculation is completed in the cloud to obtain the final inference result, thus realizing high-precision and high-responsive real-time CNN inference. It dynamically adjusts the model compression rate according to the real-time task load and network environment, and supports real-time streaming intelligent analysis in a fluctuating network environment.

[0035] The present invention uses channel pruning and unified affine quantization to jointly compress the intermediate result feature map of a given CNN layer, significantly compressing the amount of intermediate communication data with a small loss of accuracy, thereby improving the efficiency of edge-cloud collaborative CNN inference.

[0036] Furthermore, by converting a given CNN model into the corresponding SW-CNN, it supports dynamic adjustment of the compression rate of the specified layer in the model in the inference service, and then adjusts the communication volume required for collaborative reasoning to adapt to real-time changes in network conditions, thereby improving the resilience of edge-cloud collaborative CNN reasoning to actual network fluctuations.

[0037] Furthermore, based on the model predictive control algorithm, the edge-cloud collaborative CNN inference controller quickly determines and adjusts the compression rate of the SW-CNN model according to real-time network fluctuations, optimizing the responsiveness and accuracy of streaming services under given delay requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the implementation of the method in this paper;

[0039] Figure 2 This is the logic flow chart of the method in this paper; DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to the accompanying drawings.

[0041] refer to Figure 1 The present invention provides a real-time edge-cloud collaborative CNN inference method for streaming analysis, comprising the following steps:

[0042] Step 1: Convert the given CNN model into a real-time and efficient edge-cloud collaborative inference model, SWitchable-CNN (SW-CNN). On the edge device, feature map compression (Feature-Map Compression) is used to significantly compress the amount of data uploaded to the cloud with little computational overhead and accuracy loss. The compressed CNN model is then converted into the corresponding SW-CNN to support real-time adjustment of the intermediate communication compression rate. Step 1 includes the following steps:

[0043] Step 1.1, feature map compression, given that the communication data between CNN layers is a feature map, the data volume D s =F s *F n *F b , where F s Represents the area of ​​a single feature map, F n Indicates the number of feature maps, F b Indicates the number of bits of quantized data. In order to reduce the amount of transmitted data, channel pruning (ChannelPruning) and uniform affine quantization (UniformAffineQuantization) are used to reduce F n and F b , assuming that the number of feature maps remaining after the compression operation is F rn , the remaining data quantization bits are F rb , then (F rn ,F rb ) is the current compression setting, the communication compression ratio of this setting is c=1-(F rn *F rb ) / (F n *F b ), residual rate r = 1-c. After compression, the model is fine-tuned on the original dataset through end-to-end training to restore accuracy. Since different compression settings may have the same communication compression rate, the Pareto set of compression rate-accuracy is calculated to ensure that the highest accuracy solution is selected under the same compression rate.

[0044] Step 1.2, SW-CNN conversion. For a given CNN model, a large number of CNN models with different compression rates and accuracies can be obtained based on step 1.1. Since deploying a large number of CNN models at the same time will bring unacceptable storage overhead and model switching delay, the given CNN model is converted into a SW-CNN model that integrates CNN models with different compression rates to achieve real-time switching of compression rates. For a given CNN model, its corresponding SW-CNN model has the same structure. For a given CNN model with F N and F BThe partition layer can generate different sub-models according to different compression settings. Except for the given partition (compression) layer, different sub-models share all parameters. For the compression layer, different sub-models are based on (F rn ,F rb ) are nested and share parameters, using (F rn ,F rb ) The sub-models of the compression setting use numbers from 1 to F rn Feature maps and 1 to F rb The number of data bits, so (F rn ,F rb ) The large sub-model contains the sub-model with a small number of feature maps and compression bits, that is, the small sub-model is nested in the large sub-model. Step 1.2 includes the following steps:

[0045] Step 1.2.1. Model initialization: For a given CNN model, first train a CNN model with the same structure as its corresponding SW-CNN, and use its parameters to initialize the SW-CNN model corresponding to the given CNN model to reduce the average accuracy loss of the sub-model;

[0046] Step 1.2.2: Model distillation. Use the pre-trained model in step 1.2.1 as a teacher model to guide the training of the remaining sub-models to reduce the maximum accuracy loss.

[0047] Step 1.2.3: Model clustering. Based on steps 1.2.1 and 1.2.2, iteratively train the SW-CNN. In each training iteration, all sub-models independently complete their forward reasoning and backpropagation processes. The gradients are then summed and used to update the SW-CNN model. In the first epoch of training, the gradients of different sub-models are clustered using DBSCAN to group the sub-models. In subsequent training, only one model is sampled from each group during each iterative update. The number of sub-models trained is the number of sub-model groups, which is adjusted by the eps parameter in the DBSCAN algorithm.

[0048] Step 2: System Profiling. Based on step 1, information is collected on the SW-CNN corresponding to the given CNN model in the given edge cloud system (including the edge computing device layer and the cloud data center layer) to obtain the communication computing delay of each layer of the SW-CNN under each compression rate. An N-layer CNN model is deployed in a 2-layer edge cloud system, with the layer divided into l and the compression rate c. The CNN models from 0 to l layers run on the edge device, and the CNN models from l+1 to N layers run on the cloud server. Under this division and compression scheme, the end-to-end delay of CNN inference is in is the sum of the delays of the CNN model inference on the edge, Tt (l)*c is the sum of the communication delays between edge and cloud, is the sum of the inference delays of the remaining CNN models on the cloud. Different (l, c) correspond to different inference delays and accuracies. l is specified by the user. Based on step 1, the given CNN is converted to a SW-CNN. c can be selected from the compression rate-accuracy Pareto set of the SW-CNN.

[0049] Step 3: Modeling the real-time edge-cloud collaborative reasoning control problem. Based on steps 1 and 2, for the streaming reasoning tasks that arrive in the time sequence i=1,2..., adjust c according to the real-time network performance to achieve high-precision, low-latency CNN reasoning. When the previous task p n-1 When completed, the controller uses the current number of tasks p n , the remaining time left by the previous task for the current task l n-1 and the current network bandwidth b n As input, determine the current compression rate c for the batch of tasks n To maximize the real-time performance index RP, if task p n-1 In t pn-1 When the time is completed, the next batch of tasks p n Completion time where r(c n ,p n ) represents the computation time of the batch of tasks, Indicates the communication time, taking into account the user-limited delay l user , task p n The remaining time is Where (x) - =min{x,l user} to ensure that the remaining time is not greater than the time given by the user, Δt (n-1,n) is the time interval between the arrival of two batches of tasks, if l n <0Task p n There must be a delay, for tasks p1 to p n , the average precision is The delay rate is in() * represents the step function, and the control objective function is Where γ is a non-negative real number, which indicates the user's preference for real-time control. The larger γ is, the more the control tends to reduce the delay rate and can tolerate higher precision loss. The real-time reasoning control problem is modeled as solving

[0050]

[0051]

[0052] (l0=l user )

[0053] c1,c2,…,c N ∈C Pareto

[0054] Among them, C Pareto It means that only the compression rate in the Pareto set is selected to ensure the highest reasoning accuracy under the same compression rate. l0 represents the remaining time of the first task. If there is no backlog task, it can be processed immediately. l0 is equal to l user ; If there is a backlog task, and the current backlog task has arrived at time t0, then l0=l user -t0;

[0055] Step 4: Initialize the task queue;

[0056] Step 5: Real-time edge-cloud collaborative reasoning control. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on step 3 according to the current task remaining time information and real-time network conditions. It selects a suitable compression rate for SW-CNN to reduce the task delay rate and improve the reasoning accuracy. If the task queue is empty, it suspends it; otherwise, it loads the current reasoning task p from the queue. n , sample the current bandwidth b n , the remaining time of the previous task Based on the current queue length N, the model predictive control algorithm is used to predict the tasks and bandwidth at N time points in the future. Where N is equal to the number of backlog task batches. Based on step 3, a real-time edge-cloud collaborative inference control problem is modeled. The planning optimization problem is transformed into a combinatorial optimization problem through discrete bandwidth data. The simulated annealing algorithm is used to search for the optimal solution. In order to avoid the exponential explosion of the search space as the number of backlog task batches increases, the feasible strategy space attenuation mechanism is used to ensure real-time performance. When the backlog of tasks increases, a higher compression rate is selected to ensure real-time performance. Let the complete compression rate set be C Pareto When N batches of tasks are backlogged in the queue, Pareto Delete the first N smaller compression rates in ascending order to obtain the current feasible compression rate set C' Pareto , and make the capacity of the feasible set card(C' Pareto ) is equal to (card(C Pareto )-N) + , when N=1, the search space is equal to card(C Pareto ), as N increases, the capacity of the feasible set card (C' Pareto ) may gradually increase, but when N≥card(C Pareto )-1 after the capacity of the feasible set card (C' Pareto ) decays to 1, avoiding the exponential explosion of the search space as N increases;

[0057] Step 6: Based on step 5, adjust the compression rate of SW-CNN to c n ;

[0058] Step 7: Analyze task p n ;

[0059] Step 8. If not finished, repeat steps 5 to 7.

[0060] refer to Figure 2 The present invention provides a real-time edge-cloud collaborative convolutional neural network inference method for streaming analysis. Its logical architecture includes three parts: SW-CNN conversion, system information collection, and real-time edge-cloud collaborative CNN inference control, with the main parts being SW-CNN conversion and real-time edge-cloud collaborative CNN inference control. In order to improve the efficiency of edge-cloud collaborative CNN inference, channel pruning and unified affine quantization are used to jointly compress the intermediate result feature map of a given CNN layer, significantly compressing the amount of intermediate communication data with a small loss of accuracy; in order to improve the resilience of edge-cloud collaborative CNN inference to actual network fluctuations, by converting a given CNN model into a corresponding SW-CNN, it supports dynamic adjustment of the compression rate of a specified layer in the model in the inference service, thereby adjusting the communication volume required for collaborative inference to adapt to real-time changes in network conditions; in order to optimize the responsiveness and accuracy of streaming services under given delay requirements, based on the model predictive control algorithm, the edge-cloud collaborative CNN inference controller quickly determines and adjusts the compression rate of the SW-CNN model according to real-time network fluctuations.

[0061] In another embodiment of the present invention, a real-time edge-cloud collaborative convolutional neural network inference system is provided, which can be used to implement the above-mentioned real-time edge-cloud collaborative convolutional neural network inference method. Specifically, the real-time edge-cloud collaborative convolutional neural network inference system includes:

[0062] The model conversion module is used to compress the CNN model through feature maps on the edge device and convert the compressed CNN model into the edge-cloud collaborative inference model SW-CNN;

[0063] The delay calculation module is used to collect information in a given edge-cloud system to obtain the communication calculation delay of each layer of the SW-CNN at various compression rates;

[0064] The compression rate adjustment module is used to model the real-time edge-cloud collaborative CNN inference control problem and adjust the compression rate c of streaming inference tasks arriving in the time sequence i=1,2... according to the real-time network performance;

[0065] The compression rate selection module is used to initialize the task queue. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on the current task remaining time information and real-time network conditions, and selects the compression rate for SW-CNN; adjusts the compression rate of SW-CNN to cn , analysis task p n .

[0066] The present invention solves the problem that existing technologies cannot efficiently process streaming CNN inference tasks in an actual fluctuating network environment. The CNN model is divided into two parts: the edge and the cloud. A small amount of model calculations are performed on the edge to compress the original data. The intermediate data of the CNN model output by the edge is then uploaded to the cloud. A large amount of model calculations are completed in the cloud to obtain the final inference results, achieving high-precision and high-responsiveness real-time CNN inference. The model compression rate is dynamically adjusted according to the real-time task load and network environment, supporting real-time streaming intelligent analysis in a fluctuating network environment. The present invention can ensure both efficient processing of streaming CNN inference tasks and the resilience of real-time edge-cloud collaborative CNN inference to network fluctuations.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A real-time edge-cloud collaborative convolutional neural network inference method, characterized by: The following steps are involved: On the edge device, the CNN model is compressed by feature map and converted into the edge-cloud collaborative inference model SW-CNN. Information collection is performed in a given edge-cloud system to obtain the communication computation delay of each layer of SW-CNN at various compression rates; Model the real-time edge-cloud collaborative CNN inference control problem and adjust the compression ratio c based on the real-time network performance for streaming inference tasks arriving in the time sequence i=1,2...; Initialize the task queue. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on the current task remaining time information and real-time network conditions. It selects the compression rate for SW-CNN and adjusts the compression rate of SW-CNN to c n , process task p n ; Task p n After completion, if the task queue is not empty, the compression rate of SW-CNN is repeatedly adjusted to c n , process task p n , until all tasks in the task queue are processed; The CNN model is compressed through feature maps on edge devices, including: Given that the communication data between CNN layers is a feature map, the amount of data D s =F s *F n *F b , where F s Represents the area of ​​a single feature map, F n Indicates the number of feature maps, F b Indicates the number of bits of quantized data, respectively reducing F n and F b , assuming that the number of feature maps remaining after the compression operation is F rn , the remaining data quantization bits are F rb , then (F rn ,F rb ) is the current compression setting, the communication compression ratio of this setting is c=1-(F rn *F rb ) / (F n *F b ), with a residual rate of r = 1-c. After compression, the model is fine-tuned on the original dataset through end-to-end training to restore accuracy. The Pareto set of compression rate-accuracy is calculated, and the solution with the highest accuracy at the same compression rate is selected; Converting the compressed CNN model into the edge-cloud collaborative inference model SW-CNN specifically includes: SW-CNN conversion, for a given CNN model, obtains CNN models with different compression rates and accuracy, converts the given CNN model into a SW-CNN model that integrates CNN models with different compression rates. For a given CNN model, its corresponding SW-CNN model has the same structure. For a given CNN model with F N and F B The partition layer generates different sub-models according to different compression settings. Except for the given partition layer, different sub-models share all parameters. For the compression layer, different sub-models are based on (F rn ,F rb ) are nested and share parameters, using (F rn ,F rb ) The sub-models of the compression setting use numbers from 1 to F rn Feature maps and 1 to F rb The number of data bits, so (F rn ,F rb ) The large sub-model contains the sub-model with a small number of feature maps and compression bits, that is, the small sub-model is nested in the large sub-model; Adjust the compression ratio: When the previous task p n-1 When completed, the controller uses the current number of tasks p n , the remaining time left by the previous task for the current task l n-1 and the current network bandwidth b n As input, determine the current compression rate c for the batch of tasks n To maximize the real-time performance index RP, if task p n-1 exist When the time is completed, the next batch of tasks p n Completion time where r(c n ,p n ) represents the computation time of the batch of tasks, Indicates the communication time, taking into account the user-limited delay l user , task p n The remaining time is where (x) = min{x,l user } to ensure that the remaining time is not greater than the time given by the user, Δt (n-1,n) is the time interval between the arrival of two batches of tasks, if l n <0Task p n There must be a delay, for tasks p1 to p n , the average precision is The delay rate is in() * represents the step function, and the control objective function is Where γ is a non-negative real number, which indicates the user's preference for real-time control. The larger γ is, the more the control tends to reduce the delay rate and can tolerate higher precision loss. The real-time reasoning control problem is modeled as solving s.t. (l0＝l user ) c1,c2,…,c N ∈C Pareto Among them, C Pareto It means that only the compression rate in the Pareto set is selected to ensure the highest reasoning accuracy under the same compression rate. l0 represents the remaining time of the first task. If there is no backlog task, it can be processed immediately. l0 is equal to l user ; If there is a backlog task, and the current backlog task has arrived at time t0, then l0=l user -t0; Choosing compression rate for SW-CNN If the task queue is empty, suspend; otherwise, load the current inference task p from the queue n , sample the current bandwidth b n , the remaining time of the previous task Based on the current queue length N, the task and bandwidth at N future time points are predicted based on the model predictive control algorithm. Where N is equal to the number of backlogged task batches. We model the real-time edge-cloud collaborative reasoning control problem. We transform the planning optimization problem into a combinatorial optimization problem by using discrete bandwidth data. We use the simulated annealing algorithm to search for the optimal solution. When the backlog of tasks increases, we select a higher compression rate to ensure real-time performance. Let the complete compression rate set be C. Pareto , when N batches of tasks are backlogged in the queue, Pareto Delete the first N smaller compression rates in ascending order to obtain the current feasible compression rate set C' Pareto , and make the capacity of the feasible set card(C' Pareto ) is equal to (card(C Pareto )-N) + , when N=1, the search space is equal to card(C Pareto ), as N increases, the capacity of the feasible set card (C' Pareto ) gradually increases.

2. A real-time edge-cloud collaborative convolutional neural network inference method according to claim 1, characterized in that: Channel pruning and uniform affine quantization are used to reduce F n and F b .

3. The real-time edge-cloud collaborative convolutional neural network inference method according to claim 1, characterized in that: For a given CNN model, first train a CNN model with the same structure as its corresponding SW-CNN, and use its parameters to initialize the SW-CNN model corresponding to the given CNN model; the pre-trained model is used as a teacher model to guide the training of the remaining sub-models; iteratively train SW-CNN, and in each training iteration, all sub-models independently complete their forward reasoning and backpropagation processes, and then the gradients are summed to update the SW-CNN model. At the beginning of the first epoch of training, the gradients of different sub-models are clustered using DBSCAN to group the sub-models. In subsequent training, only one model is sampled in each group for training and updating during each iterative update. The number of sub-models trained is the number of sub-model groups, which is adjusted by the eps parameter in the DBSCAN algorithm.

4. The real-time edge-cloud collaborative convolutional neural network inference method according to claim 1, characterized in that: Calculate delay: Deploy an N-layer CNN model in a 2-layer edge-cloud system, with the partitioning layer being l and the compression rate being c. CNN models from 0 to l layers run on edge devices, and CNN models from l+1 to N layers run on cloud servers. Under this partitioning and compression scheme, the end-to-end latency of CNN inference is in is the sum of the delays of the CNN model inference on the edge, T t (l)*c is the sum of the communication delays between edge and cloud, It is the sum of the inference delays of the remaining CNN models on the cloud. Different (l, c) correspond to different inference delays and accuracies. l is specified by the user. The given CNN is converted into a SW-CNN. c is selected from the compression rate-accuracy Pareto set of the SW-CNN.

5. A real-time edge-cloud collaborative convolutional neural network inference system, characterized by: include: The model conversion module is used to compress the CNN model through feature maps on the edge device and convert the compressed CNN model into the edge-cloud collaborative inference model SW-CNN; The delay calculation module is used to collect information in a given edge-cloud system to obtain the communication calculation delay of each layer of the SW-CNN at various compression rates; The compression rate adjustment module is used to model the real-time edge-cloud collaborative CNN inference control problem and adjust the compression rate c of streaming inference tasks arriving in the time sequence i=1,2... according to the real-time network performance; The compression rate selection module is used to initialize the task queue. The controller models and solves the real-time edge-cloud collaborative reasoning control problem based on the current task remaining time information and real-time network conditions, and selects the compression rate for SW-CNN; adjusts the compression rate of SW-CNN to c n , analysis task p n ; Converting the compressed CNN model into the edge-cloud collaborative inference model SW-CNN specifically includes: SW-CNN conversion, for a given CNN model, obtains CNN models with different compression rates and accuracy, converts the given CNN model into a SW-CNN model that integrates CNN models with different compression rates. For a given CNN model, its corresponding SW-CNN model has the same structure. For a given CNN model with F N and F B The partition layer generates different sub-models according to different compression settings. Except for the given partition layer, different sub-models share all parameters. For the compression layer, different sub-models are based on (F rn ,F rb ) are nested and share parameters, using (F rn ,F rb ) The sub-models of the compression setting use numbers from 1 to F rn Feature maps and 1 to F rb The number of data bits, so (F rn ,F rb ) The large sub-model contains the sub-model with a small number of feature maps and compression bits, that is, the small sub-model is nested in the large sub-model; Adjust the compression ratio: When the previous task p n-1 When completed, the controller uses the current number of tasks p n , the remaining time left by the previous task for the current task l n-1 and the current network bandwidth b n As input, determine the current compression rate c for the batch of tasks n To maximize the real-time performance index RP, if task p n-1 exist When the time is completed, the next batch of tasks p n Completion time where r(c n ,p n ) represents the computation time of the batch of tasks, Indicates the communication time, taking into account the user-limited delay l user , task p n The remaining time is Where (x) - =min{x,l user } to ensure that the remaining time is not greater than the time given by the user, Δt (n-1,n) is the time interval between the arrival of two batches of tasks, if l n <0Task p n There must be a delay, for tasks p1 to p n , the average precision is The delay rate is in() * represents the step function, and the control objective function is Where γ is a non-negative real number, which indicates the user's preference for real-time control. The larger γ is, the more the control tends to reduce the delay rate and can tolerate higher precision loss. The real-time reasoning control problem is modeled as solving s.t. (l0＝l user ) c1,c2,…,c N ∈C Pareto Among them, C Pareto It means that only the compression rate in the Pareto set is selected to ensure the highest reasoning accuracy under the same compression rate. l0 represents the remaining time of the first task. If there is no backlog task, it can be processed immediately. l0 is equal to l user ; If there is a backlog task, and the current backlog task has arrived at time t0, then l0=l user -t0; Choosing compression rate for SW-CNN If the task queue is empty, suspend; otherwise, load the current inference task p from the queue n , sample the current bandwidth b n , the remaining time of the previous task Based on the current queue length N, the task and bandwidth at N future time points are predicted based on the model predictive control algorithm. Where N is equal to the number of backlogged task batches. We model the real-time edge-cloud collaborative reasoning control problem. We transform the planning optimization problem into a combinatorial optimization problem by using discrete bandwidth data. We use the simulated annealing algorithm to search for the optimal solution. When the backlog of tasks increases, we select a higher compression rate to ensure real-time performance. Let the complete compression rate set be C. Pareto When N batches of tasks are backlogged in the queue, Pareto Delete the first N smaller compression rates in ascending order to obtain the current feasible compression rate set C' Pareto , and make the capacity of the feasible set card(C' Pareto ) is equal to (card(C Pareto )-N) + , when N=1, the search space is equal to card(C Pareto ), as N increases, the capacity of the feasible set card (C' Pareto ) gradually increases.

Images