An edge-assisted multi-terminal model online update method

Abstract

The present invention relates to an edge-assisted multi-terminal model online updating method, which belongs to the field of video analysis technology. First, the video data is screened, and then the redundant video frames are filtered. The terminal uploads the finally screened video frames to the edge server. On the edge server side, the video frames are detected by deploying an uncompressed target detection model, and its detection results are used as pseudo labels of these video frames. The pseudo labels with high confidence are selected as the data set for model retraining according to the classification confidence scores of the pseudo labels. A task scheduling algorithm is used to reasonably schedule the execution order of multiple tasks and allocate appropriate computing and video memory resources. When the retraining is completed, the edge server transmits the terminal model parameters back to the terminal and replaces the terminal compression model. The terminal uses the updated model for subsequent video analysis.

Description

Technical Field

[0001] The present invention belongs to the field of video analysis technology, and relates to retraining of compressed target detection models and task scheduling and resource allocation among multiple training tasks, and in particular to online learning of edge-assisted multi-terminal compression models. Background Art

[0002] In recent years, most mobile applications, such as mobile VR / AR and self-driving cars, require real-time video analysis. Video analysis tasks, such as object detection and instance segmentation, rely heavily on deep neural networks. Offloading deep neural networks to cloud or edge servers for inference suffers from high latency and privacy concerns, making it unsuitable for real-time video analysis applications. Deploying deep learning models for inference on end devices has attracted significant attention due to their lower inference latency. However, deploying large deep learning models on resource-limited mobile devices is difficult.

[0003] To address this issue, researchers have designed various model compression methods to reduce the computational and storage consumption of deep neural networks. However, due to the sparsity of model parameters, compressed deep learning models deployed on devices suffer from data drift. This is when the real-time video data captured by the device deviates from the data used to train the model, violating the IID assumption. This can significantly reduce model inference accuracy.

[0004] Online learning is a practical and promising solution to the data drift problem. This involves retraining compression models using real-time video data streaming into end devices. While existing online learning efforts have made significant progress, none accurately considers the impact of changing mobile scenarios on the accuracy degradation of compressed deep learning models. Furthermore, edge server GPU computing and memory resources can easily become bottlenecks for online learning of multi-endpoint compression models, increasing the time required for end-point model online learning and impacting the overall inference accuracy of the system. Summary of the Invention

[0005] Technical problems to be solved

[0006] In order to avoid the deficiencies of the prior art, the present invention provides an edge-assisted multi-terminal model online updating method.

[0007] Technical Solution

[0008] An edge-assisted multi-terminal model online update method is characterized by the following steps:

[0009] Step 1: While using the compressed object detection model for real-time video analysis, the terminal device will filter the video data in the real scene and select video frames for terminal model retraining. Specifically, because the difference in feature distribution between the real-time video stream and the model training data, as well as the scene variation factor, has a significant impact on model accuracy, a scene change measurement metric d is defined:

[0010]

[0011]

[0012] Among them, V c represents the difference in feature distribution between the video stream and the model training data, a represents the average pixel value outside the target in the video frame, and Represent the data features of model training data and real-time video stream respectively, Indicates the calculation of the first-order moment, σ(·) indicates the calculation of the standard deviation, and ||·||2 indicates the calculation of the L2 norm;

[0013] Filter out video frames with scene change d below a threshold for retraining the terminal model;

[0014] Step 2: To further reduce the time for online learning, for the video frames selected in step 1, redundant video frames are first filtered out. Then, a video frame selection algorithm based on video content is used to select the video frames that contribute the most to the accuracy gain of model retraining and upload them to the server for subsequent online model learning.

[0015] Step 3: The terminal uploads the final filtered video frames to the edge server. On the edge server, an uncompressed object detection model is deployed to detect the video frames and the detection results are used as pseudo labels for these video frames for subsequent retraining of the compression model.

[0016] Step 4: During the terminal model retraining process, pre-generate the optimal quantization strategy for different terminal devices. Based on the obtained quantization strategy, evaluate the redundancy of each layer of the compression model and freeze the network layers with redundancy exceeding the set threshold so that they do not participate in training.

[0017] Step 5: Predict the video memory required for model retraining for this task by calculating the model parameters, intermediate results, backpropagation gradients, optimizer, and the video memory resources occupied by the underlying library. Use video frames and several training cycles to fit a nonlinear curve between training cycle and prediction accuracy. Use a lightweight three-layer neural network to construct the impact of terminal retraining time and terminal model retraining features. Using model size, retraining data volume, number of model layers involved in retraining, retraining cycle, and batch size as input, the retraining time of the terminal model is calculated.

[0018] Step 6: Based on the retraining task information predicted in step 5, a scheduling algorithm is used to select the appropriate task to be executed on the GPU;

[0019] Step 7: Allocate resources to multiple tasks in the GPU; for memory resources, allocate sufficient memory resources according to the memory prediction in the task attributes; for computing resources, the computing resources C obtained by the i-th task i for:

[0020]

[0021] Where N is the number of tasks running on the GPU, C s (t) represents the available computing resources of the GPU; t ri Indicates the retraining time required for the terminal model;

[0022] Step 8: After the retraining is completed, the edge server transmits the terminal model parameters back to the terminal and replaces the terminal compression model. The terminal uses the updated model for subsequent video analysis.

[0023] A further technical solution of the present invention: Step 2 is specifically as follows: first, the pixel difference of the video frames is calculated using the frame difference method, and the similarity between the video frames is evaluated based on this. For video frames whose pixel difference is less than a certain threshold, they are determined to be redundant frames and are filtered out; next, the contribution of each video frame to the model retraining is predicted through the global features of the data distribution, and the non-redundant alternative frames are further screened.

[0024] A further technical solution of the present invention: Step 6 is specifically as follows: all tasks are divided into k queues according to the accuracy gain from large to small, the tasks in each queue are sorted from small to large according to the training time, and the tasks of the first echelon are placed in the GPU in sequence until the remaining video memory of the GPU is insufficient to perform the tasks of the first echelon; whenever a task is completed, a task is selected from the first echelon and placed in the GPU for execution; when all tasks in the first echelon are completed, the remaining echelons are moved up, and the nth queue becomes the n-1th queue; when a new task arrives at the server asynchronously, if the model accuracy corresponding to the task is lower than the minimum threshold, it is directly placed in the first queue for waiting; if the model accuracy corresponding to the task is higher than the minimum threshold, considering the task waiting time, the task is placed in the next queue of its corresponding queue.

[0025] A computer system, characterized in that it includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned method.

[0026] A computer-readable storage medium is characterized by storing computer-executable instructions, which are used to implement the above method when executed.

[0027] Beneficial effects

[0028] This paper provides an edge-assisted online multi-terminal model update method that utilizes content-aware frame selection, pseudo-label filtering, and frozen training to achieve rapid and efficient updates of individual models. It also maximizes the average inference accuracy across all terminal devices through task scheduling and an adaptive resource allocation algorithm. This method addresses the data drift issues faced by compressed deep models in the face of frequent scene changes on mobile devices, overcomes the computational and memory resource bottlenecks faced by edge servers in multitasking scenarios, effectively improves model accuracy, and maximizes the average inference accuracy of all terminal models. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like parts throughout the drawings.

[0030] Figure 1 A schematic diagram of the structure of an edge-assisted multi-terminal model online update in an example of the present invention;

[0031] Figure 2 This is a schematic diagram of asynchronous task scheduling;

[0032] Figure 3 Schematic diagram of asynchronous task scheduling algorithm. DETAILED DESCRIPTION

[0033] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0034] The present invention utilizes the following principle: the average inference accuracy of the terminal model is closely related to the model update delay. The lower the model update delay, the sooner the new high-precision model can be deployed and put into operation, and thus the average inference accuracy of the terminal model is higher. The update delay is composed of data upload time, task scheduling time, model retraining time, and parameter return time. Reducing the amount of uploaded data and the number of training parameters required for the model can effectively reduce the model update delay. The feature distribution difference between the real-time video stream and the model training data in the mobile scenario and the scene variation factors in the mobile scenario (such as background) have a very large impact on the accuracy of the compressed deep model. Therefore, they can be used as indicators for screening data to select representative data for upload and reduce the amount of uploaded data. At the same time, freezing irrelevant and redundant layers during the model retraining process can reduce the amount of training parameters. As the number of models participating in the online model update increases, the GPU computing and memory resources of the edge server become bottlenecks. We use task scheduling and adaptive resource allocation to achieve fair resource allocation between multiple tasks, thereby maximizing the average inference accuracy of all terminal models.

[0035] There are 3 drawings in this invention, please refer to Figure 1 , Figure 2 , Figure 3 As shown, the specific steps of the present invention are as follows:

[0036] Step 1: While the terminal device uses the compressed terminal object detection model for real-time video analysis, it also filters the video data in the real scene and selects video frames for terminal model retraining. Specifically, because the difference in feature distribution between the real-time video stream and the terminal model training data, as well as the scene variation factor, has a significant impact on the accuracy of the terminal model, we define the scene variation metric d:

[0037]

[0038]

[0039] Among them, V c represents the difference in feature distribution between the video stream and the terminal model training data for category c, a represents the average pixel value outside the target in the video frame, and Represent the data features of the terminal model training data and the real-time video stream, Denotes the calculation of the first-order moment, σ(·) denotes the calculation of the standard deviation, and ||·||2 denotes the calculation of the L2 norm. When the scene change d is below a threshold, the terminal model has poor inference accuracy on these video frames. Therefore, we filter out such video frames for retraining the terminal model.

[0040] Step 2: To further reduce the time for online learning, we first filter out redundant video frames for the video frames selected in Step 1. We then use a video content-based video frame selection algorithm to select the video frames that contribute most to the accuracy gain of the terminal model retraining and upload them to the server for subsequent online learning of the terminal model. Specifically, we first use the frame difference method to calculate the pixel difference of the video frames, based on which we evaluate the similarity between video frames. For video frames with pixel differences less than a certain threshold, we determine them to be redundant frames and filter them out. Next, we use the global characteristics of the data distribution to predict the contribution of each video frame to the terminal model retraining, and further screen non-redundant candidate frames.

[0041] Step 3: The terminal uploads the final filtered video frames to the edge server. The golden model (uncompressed model) located on the edge server annotates the video frames and uses the pseudo-label filter at the bounding box level to filter the annotated data and retrain using the high-confidence labeled bounding boxes.

[0042] Step 4: During model retraining, we pre-generate optimal quantization strategies offline for different end devices. Based on the resulting quantization strategies, we evaluate the redundancy of each layer in the end compression model. By freezing network layers with redundancy exceeding a set threshold, they are excluded from training. This reduces the end model retraining time while minimizing accuracy loss.

[0043] Step 5: As the number of end devices requesting model retraining from the edge server increases, GPU computing and memory resources reach a bottleneck. To maximize the average inference accuracy of all end models, we designed a task scheduling algorithm to schedule the execution order of multiple retraining tasks and allocate appropriate resources to them. To enrich the information about end device retraining requests, we predict the memory requirements, accuracy gain, and retraining time for each retraining task before scheduling. Specifically, we first estimate the memory required for end model retraining by calculating the end model parameters, intermediate results, backpropagation gradients, optimizer, and underlying library usage. Then, we use a least squares function to fit a nonlinear curve between training epochs and inference accuracy based on a small amount of training data and the accuracy of several training epochs, thereby predicting the accuracy improvement after end model retraining. Finally, we use a lightweight three-layer neural network to construct a prediction model that correlates end model retraining time and model features. This prediction model takes the number of model parameters, the amount of retraining data, the number of network layers involved in retraining, the number of retraining epochs, and the batch size as inputs, and outputs the end model retraining time.

[0044] Step 6: Based on the retraining task information predicted in step 5, we use a scheduling algorithm to select appropriate tasks and put them into the GPU for execution. Specifically, we divide all tasks into k queues according to the accuracy gain from large to small, and sort the tasks in each queue from small to large according to the training time. We put the tasks of the first echelon into the GPU in sequence until the remaining video memory of the GPU is insufficient to perform the tasks of the first echelon. Whenever a task is completed, a task is selected from the first echelon and put into the GPU for execution. When all tasks in the first echelon are completed, the remaining echelons move up, for example, the nth queue becomes the n-1th queue. When a new task arrives at the server asynchronously, if the model accuracy corresponding to the task is lower than the minimum threshold, it will be directly placed in the first queue to wait. If the model accuracy corresponding to the task is higher than the minimum threshold, considering the task waiting time, we will put the task into the next queue of its corresponding queue.

[0045] Step 7: We allocate resources for multiple tasks in the GPU. For memory resources, we allocate sufficient memory resources according to the memory prediction in the task attributes; for computing resources, in order to shorten the retraining time of the retraining task, we allocate more computing resources to the task with the longest retraining time, that is, the computing resources C obtained by the i-th task i for

[0046]

[0047] Where N is the number of tasks running in the GPU, Cs(t) represents the available computing resources of the GPU, and tri represents the retraining time required for the terminal model.

[0048] Step 8: After the terminal model is retrained, the edge server transmits the model parameters back to the terminal and replaces the terminal compression model. The terminal uses the updated model for subsequent video analysis.

[0049] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present invention, and these modifications or replacements should all be included in the scope of protection of the present invention.

Claims

1. An edge-assisted multi-terminal model online update method, characterized in that Here are the steps: Step 1: While using the compressed object detection model for real-time video analysis, the terminal device will filter the video data in the real scene and select video frames for terminal model retraining. Specifically, because the difference in feature distribution between the real-time video stream and the model training data, as well as the scene variation factor, has a significant impact on model accuracy, a scene change measurement metric d is defined: Among them, V c represents the difference in feature distribution between the video stream and the model training data, a represents the average pixel value outside the target in the video frame, and Represent the data features of model training data and real-time video stream respectively, Indicates the calculation of the first-order moment, σ(·) indicates the calculation of the standard deviation, and ||·||2 indicates the calculation of the L2 norm; Filter out video frames with scene change d below a threshold for retraining the terminal model; Step 2: To further reduce the time for online learning, for the video frames selected in step 1, redundant video frames are first filtered out. Then, a video frame selection algorithm based on video content is used to select the video frames that contribute the most to the accuracy gain of model retraining and upload them to the server for subsequent online model learning. Step 3: The terminal uploads the final filtered video frames to the edge server. On the edge server, an uncompressed object detection model is deployed to detect the video frames and the detection results are used as pseudo labels for these video frames for subsequent retraining of the compression model. Step 4: During the terminal model retraining process, pre-generate the optimal quantization strategy for different terminal devices. Based on the obtained quantization strategy, evaluate the redundancy of each layer of the compression model and freeze the network layers with redundancy exceeding the set threshold so that they do not participate in training. Step 5: Predict the video memory required for model retraining for this task by calculating the model parameters, intermediate results, backpropagation gradients, optimizer, and the video memory resources occupied by the underlying library. Use video frames and several training cycles to fit a nonlinear curve between training cycle and prediction accuracy. Use a lightweight three-layer neural network to construct the impact of terminal retraining time and terminal model retraining features. Using model size, retraining data volume, number of model layers involved in retraining, retraining cycle, and batch size as input, the retraining time of the terminal model is calculated. Step 6: Based on the retraining task information predicted in step 5, a scheduling algorithm is used to select the appropriate task to be executed on the GPU; Step 7: Allocate resources to multiple tasks in the GPU; for memory resources, allocate sufficient memory resources according to the memory prediction in the task attributes; for computing resources, the computing resources C obtained by the i-th task i for: Where N is the number of tasks running on the GPU, C s (t) represents the available computing resources of the GPU; t ri Indicates the retraining time required for the terminal model; Step 8: After the retraining is completed, the edge server transmits the terminal model parameters back to the terminal and replaces the terminal compression model. The terminal uses the updated model for subsequent video analysis.

2. The edge-assisted multi-terminal model online update method according to claim 1, characterized in that: Specifically, step 2 comprises the following steps: first, the pixel difference of the video frames is calculated using the frame difference method, and based on this, the similarity between the video frames is evaluated. For video frames whose pixel difference is less than a certain threshold, they are determined to be redundant frames and filtered out; next, the contribution of each video frame to the model retraining is predicted through the global features of the data distribution, and non-redundant candidate frames are further screened.

3. The edge-assisted multi-terminal model online update method according to claim 1, characterized in that: The step 6 is specifically as follows: all tasks are divided into k queues according to the accuracy gain from large to small, the tasks in each queue are sorted from small to large according to the training time, and the tasks of the first echelon are placed in the GPU in sequence until the remaining video memory of the GPU is insufficient to perform the tasks of the first echelon; whenever a task is completed, tasks are selected from the first echelon and placed in the GPU for execution; when all tasks in the first echelon are completed, the remaining echelons are moved up, and the nth queue becomes the n-1th queue; when a new task arrives at the server asynchronously, if the model accuracy corresponding to the task is lower than the minimum threshold, it is directly placed in the first queue for waiting; if the model accuracy corresponding to the task is higher than the minimum threshold, considering the task waiting time, the task is placed in the next queue of its corresponding queue.

4. A computer system, characterized in that include: One or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method of claim 1.

5. A computer-readable storage medium, characterized in that Computer-executable instructions are stored, and when the instructions are executed, they are used to implement the method of claim 1.

Images