A video frame self-adaptive offloading method and device in a cloud edge scenario

Abstract

The application relates to the technical field of cloud edge end task coordination scheduling, and discloses a video frame adaptive offloading method and device in a cloud edge end scene, which comprises the following steps: dividing a to-be-offloaded video stream into multiple video blocks according to preset time slots; obtaining a prediction accuracy based on coarse-grained features of each image frame in the video blocks; constructing a maximum theoretical accuracy expression of each video block, a time slot load queue and a time slot delay queue based on a cloud end model, an edge model and performance data of the system; further constructing a Lyapunov function and calculating a drift term; obtaining an optimization objective function based on the drift term and the maximum theoretical accuracy expression, and taking the reciprocal of the optimization objective function as a fitness function; taking different cloud offloading ratios and edge offloading ratios of each video block as individuals, obtaining target individuals of each video block by using a genetic algorithm, and offloading all the image frames in each video block to the cloud end, the edge or the local in a descending order of prediction accuracy.

Description

Technical Field

[0001] This invention relates to the field of cloud-edge-device task collaborative scheduling technology, and in particular to a video frame adaptive offloading method and apparatus in a cloud-edge-device scenario. Background Technology

[0002] Edge video analytics significantly reduces response latency by moving computational tasks or models from cloud servers to the network edge, providing crucial technical support for applications with high real-time requirements, such as intelligent transportation and medical monitoring. Depending on the scenario requirements, edge video analytics can adjust different pipeline knobs (video configurations), including video resolution, sampling frame rate, and video stream bitrate. However, when facing highly complex analysis scenarios, the detection accuracy of models deployed on resource-constrained edge devices inevitably decreases significantly, making it difficult to meet the accuracy requirements of practical applications simply by adjusting video configurations.

[0003] Collaborative cloud-edge processing has become an effective solution, but its implementation faces significant challenges. First, limited by fluctuating bandwidth resources between the edge and cloud, it's difficult to dynamically determine the number of video frames each collaborating end can process. Too many offloaded frames lead to transmission congestion and a surge in latency, potentially causing system crashes due to excessive load on devices. Too few, on the other hand, fail to fully utilize the powerful computing capabilities of the cloud to improve accuracy. Therefore, the offload scheduling algorithm needs to comprehensively consider accuracy, latency, and load to dynamically adjust the offload ratio. Second, video content is dynamically changing; the same model performs differently in different scenes. Even within the same scene, the detection difficulty of different frames varies considerably due to target movement, background changes, and other factors. Accurately identifying the most complex frames requiring cloud assistance from the video stream is crucial for efficient offloading; this requires the detection algorithm to possess frame-level complexity awareness and filtering capabilities.

[0004] Most existing offloading solutions attempt to optimize other knobs while ignoring the impact of frame rate. Some solutions utilize collaborative architecture analysis but fail to dynamically adjust the amount of offloading tasks based on real-time changes in bandwidth load, relying solely on setting a fixed sampling frame rate or manually adjusting the detection frequency to balance cloud-edge load. Only a few methods consider dynamically implementing frame-level scheduling of video content, and these methods mainly employ fine-grained convolution operations to extract image content. These methods are resource-intensive and will affect the real-time requirements of video analysis.

[0005] In summary, existing offloading methods cannot achieve a stable balance between accuracy, latency, and resource consumption while simultaneously implementing frame-level scheduling, resulting in poor scheduling accuracy and low efficiency. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem that the existing technology cannot achieve a stable balance between accuracy, latency and resource consumption while realizing frame-level scheduling, resulting in poor scheduling accuracy and low efficiency.

[0007] To address the aforementioned technical problems, this invention provides a video frame adaptive offloading method in a cloud-edge-device scenario, comprising:

[0008] The video stream to be unloaded is divided into multiple video blocks according to preset time slots;

[0009] The prediction accuracy of each image frame is obtained based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its previous image frame in the video block.

[0010] Based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame, the maximum theoretical accuracy expression for each video block is constructed.

[0011] Based on cloud model processing capabilities, edge model processing capabilities, time slot load data, and the length of a preset time slot, a time slot load queue is constructed; based on the preset time slot delay and preset deadline, a time slot delay queue is constructed; based on the time slot load queue and the time slot delay queue, a Lyapunov function is constructed, and the drift term is calculated and obtained.

[0012] Based on the drift term and the maximum theoretical accuracy expression, the optimization objective function in draft-plus-penalty form is obtained, and the negative of the optimization objective function is taken as the fitness function;

[0013] Using different cloud offloading and edge offloading ratios of each video block as individuals, an initial population is constructed and updated until the fitness function converges to obtain the target individuals of each video block.

[0014] Based on the target cloud offloading ratio and target edge offloading ratio corresponding to the target individuals in each video block, all image frames in each video block are offloaded to the cloud, edge, or local in order of prediction accuracy from largest to smallest.

[0015] Preferably, the prediction accuracy of each image frame is obtained based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its predecessor in the video block, including:

[0016] Get the first video block The gradient of each pixel in each image frame is used to identify edge pixels, and those pixels with gradients greater than a preset gradient threshold are identified as edge pixels. ;

[0017] Calculate the first video block Image entropy of one image frame , is represented as: ;

[0018] Calculate the first video block Standard deviation of optical flow between each image frame and its previous image frame , is represented as: ;

[0019] The video block Image frames , and Input the data into the trained support vector regression model to obtain the first data segment in the video block. Prediction accuracy per image frame;

[0020] in, Indicates the first video block The grayscale value in each image frame is The normalized probability of pixel occurrence. Indicates the first image frame The size of the optical flow vector per pixel This represents the average value of all optical flow vectors in the image frame. , This represents the total number of pixels in an image frame.

[0021] Preferably, based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame, a maximum theoretical accuracy expression for each video block is constructed, including:

[0022] Cloud-based model inference accuracy With the characterization of the first In the first time slot Binary parameters indicating whether an image frame is unloaded from the cloud. The product of , to obtain the first In the first time slot Theoretical accuracy of each image frame in the cloud , is represented as: ;

[0023] Inference accuracy based on edge model With the characterization of the first In the first time slot Binary parameters for whether an image frame is unloaded at the edge The product of , to obtain the first In the first time slot Edge theoretical accuracy of each image frame , is represented as: ;

[0024] Using the target tracking decay mode, based on the first In the first time slot Theoretical accuracy of each image frame in the cloud Edge theory accuracy Local theoretical accuracy and characterizing the first In the first time slot Binary parameters indicating whether an image frame is unloaded locally. Calculate the first In the first time slot Local theoretical accuracy of each image frame , is represented as: ;

[0025] Based on the Video blocks in each time slot Calculate the maximum theoretical accuracy of the video block by considering the cloud-based theoretical accuracy, edge-based theoretical accuracy, and local theoretical accuracy of each image frame. , is represented as: ;

[0026] in, , This indicates the total number of image frames in the video block; This indicates the preset accuracy attenuation coefficient. This indicates the video frame rate of the video stream to be unloaded.

[0027] Preferably, based on cloud model processing capabilities, edge model processing capabilities, time slot load data, and the length of a preset time slot, a time slot load queue is constructed, including:

[0028] Based on cloud-based model processing capabilities Edge model processing capabilities and the length of the preset time slot The amount of data processed per time slot by the computing system , is represented as: ;

[0029] Based on the system's processing capacity per time slot ,as well as Slot offload data volume Construct a time-slot load queue, represented as:

[0030] ;

[0031] in, Indicates the first System load factor at the end of the time slot Indicates the first System load factor at the end of the time slot This indicates taking the maximum value.

[0032] Preferably, a time slot delay queue is constructed based on the delay of a preset time slot and a preset deadline, including:

[0033] Based on cloud uninstallation ratio The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With cloud bandwidth Calculate cloud latency, expressed as: ;

[0034] Based on edge unloading ratio The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With edge bandwidth The edge delay is calculated and expressed as: ;

[0035] Summing the cloud latency and edge latency, we obtain the latency caused by unloading data within a preset time slot. , is represented as: ;

[0036] Based on the delay and preset deadline of the preset time slot Construct a time slot delay queue, represented as:

[0037] ;

[0038] Among them, frame parameters These are parameters representing the width, height, and data volume of an image frame. Indicates the first The system delay factor at the end of the time slot, Indicates the first The system delay factor at the end of the time slot, This indicates taking the maximum value.

[0039] Preferably, a Lyapunov function is constructed based on the time slot load queue and the time slot delay queue. , is represented as:

[0040] ;

[0041] in, Indicates the first State variables of time slots The corresponding Lyapunov function, Indicates the first System load factor at the end of the time slot The square of, Indicates the first System delay factor at the end of the time slot The square of.

[0042] Preferably, the drift term is calculated and obtained, expressed as:

[0043] ;

[0044] in, Indicates the first The drift term of each time slot, It represents conditional expectation.

[0045] Preferably, based on the drift term and the maximum theoretical accuracy expression, the optimization objective function in draft-plus-penalty form is obtained, expressed as:

[0046] ;

[0047] in, Indicates the maximum theoretical accuracy. This represents the penalty coefficient.

[0048] Preferably, based on the target cloud offloading ratio and target edge offloading ratio corresponding to the target individuals in each video block, all image frames in each video block are offloaded to the cloud, edge, or local storage in descending order of prediction accuracy, including:

[0049] Sort all image frames in the video block in descending order of prediction accuracy to obtain the unloading sequence;

[0050] The first in the sequence to be unloaded The frame image is unloaded to the cloud and removed from the unloading sequence, and the unloading sequence is updated.

[0051] The first in the sequence to be unloaded The frame image is unloaded to the edge and removed from the unloading sequence, and the unloading sequence is updated.

[0052] Unload the remaining image frames in the sequence to be unloaded to the local machine;

[0053] in, This indicates the percentage of data uninstalled from the cloud. This indicates the video frame rate of the video stream to be unloaded. This indicates the percentage of data unloaded at the edge.

[0054] This embodiment also provides an apparatus based on the video frame adaptive offloading method in the cloud-edge-device scenario described above, comprising:

[0055] The video segmentation module is used to divide the video stream to be unloaded into multiple video blocks according to preset time slots;

[0056] The prediction accuracy acquisition module is used to obtain the prediction accuracy of each image frame based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its previous image frame in the video block.

[0057] The unloading ratio acquisition module is used to construct the maximum theoretical accuracy expression for each video block based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame; construct a time slot load queue based on the processing capacity of the cloud model, the processing capacity of the edge model, the time slot load data, and the length of the preset time slot; construct a time slot delay queue based on the delay and the preset deadline of the preset time slot; construct a Lyapunov function based on the time slot load queue and the time slot delay queue, and calculate and obtain the drift term; based on the drift term and the maximum theoretical accuracy expression, transform and obtain the draft-plus-penalty form of the optimization objective function, and take the negative of the optimization objective function as the fitness function; construct an initial population with different cloud unloading ratios and edge unloading ratios of each video block as individuals, update it, and obtain the target individuals for each video block until the fitness function converges;

[0058] The adaptive unloading module is used to unload all image frames in each video block to the cloud, edge, or local machine in descending order of prediction accuracy, based on the target cloud unloading ratio and target edge unloading ratio corresponding to the target individuals in each video block.

[0059] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0060] The video frame adaptive offloading method in the cloud-edge-device scenario described in this invention obtains the prediction accuracy of each image frame based on the number of edge pixels, image entropy, and the standard deviation of optical flow between each image frame and its predecessor. It achieves accurate and lightweight quantification of the detection difficulty of each frame based on the coarse-grained features of the image frames, completing frame complexity identification without performing fine-grained convolution operations. This significantly reduces the computational resource consumption of edge devices, avoids the impact of feature extraction on the real-time performance of video analysis, adapts to scenarios with limited edge device resources, and provides a standardized basis for subsequent cloud, edge, or local scheduling allocation based on accuracy, achieving frame-level precise scheduling.

[0061] This invention applies Lyapunov optimization theory to design an algorithm that can dynamically adjust the proportion of offloaded frames based on system state. It constructs a time-slot load queue by combining the processing capabilities of the cloud and edge models, time-slot load data, and preset time-slot lengths to constrain the hardware processing load of the cloud and edge to within limits. It also constructs a time-slot delay queue by combining time-slot delay and preset deadlines to constrain end-to-end analysis latency to not exceed business requirements. The feasible domain for the offload ratio is defined from the two core dimensions of load and latency, avoiding transmission congestion and system crashes caused by excessive offloading, or wasted cloud computing power caused by insufficient offloading. A Lyapunov function is constructed based on the load queue and virtual queue, and a drift term is calculated to characterize the degree to which the current load and latency deviate from the stable state; the smaller the drift term, the more stable the system. Finally, the maximum theoretical accuracy expression, constructed from the inference accuracy of the cloud model, the inference accuracy of the edge model, and the frame offload location parameters, is transformed into an optimization objective function in the form of a draft-plus-penalty, achieving dual-objective optimization of system stability and analysis accuracy with low load and low latency. Meanwhile, using the cloud offloading ratio and the edge offloading ratio as optimization variables, the optimal solution of the objective function is obtained through a genetic algorithm. This yields the target cloud offloading ratio and the target edge offloading ratio for each video block. This ratio can adapt to the bandwidth, load, and latency status of the current time slot, dynamically determining the number of frames that each video block needs to be offloaded to the cloud or the edge. This achieves accurate matching of the number of offloaded frames in bandwidth fluctuation scenarios, thereby achieving a long-term balance between accuracy, latency, and load, and ensuring the stability of system operation. Attached Figure Description

[0062] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0063] Figure 1 This is a flowchart of the video frame adaptive offloading method in the cloud-edge-device scenario of the present invention;

[0064] Figure 2 This is a flowchart of the steps to obtain prediction accuracy;

[0065] Figure 3 This is a flowchart illustrating the steps for obtaining the target cloud offloading ratio and the target edge offloading ratio.

[0066] Figure 4 This is a flowchart of the frame strategy generation steps;

[0067] Figure 5 This is a flowchart of the theoretical accuracy calculation achieved under the frame strategy;

[0068] Figure 6 This is a schematic diagram illustrating the principle of the adaptive video frame offloading method in a cloud-edge-device scenario. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0070] Reference Figure 1 The flowchart shown is a step-by-step flowchart of the video frame adaptive offloading method in the cloud-edge-device scenario of the present invention, and the specific steps include S101 to S107.

[0071] S101: Divide the video stream to be unloaded into multiple video blocks according to preset time slots.

[0072] S102: Based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its predecessor in the video block, obtain the prediction accuracy of each image frame, including:

[0073] S102-1: Retrieve the first video block Image frames Each pixel gradient Pixels with gradients greater than a preset gradient threshold are identified as edge pixels, and the number of edge pixels is obtained. ;

[0074] S102-2: Calculate the number of segments in the video block. Image entropy of one image frame , is represented as: ;

[0075] S102-3: Calculate the first... in the video block Standard deviation of optical flow between each image frame and its previous image frame , is represented as: ;

[0076] S102-4: The video block contains the... Image frames , and Input the data into the trained support vector regression model to obtain the first data segment in the video block. Prediction accuracy per image frame;

[0077] in, Indicates the first video block The grayscale value in each image frame is The normalized probability of pixel occurrence. Indicates the first image frame The size of the optical flow vector per pixel This represents the average value of all optical flow vectors in the image frame. , This represents the total number of pixels in an image frame.

[0078] Reference Figure 2 The diagram shows the flowchart for obtaining prediction accuracy. Specifically, the input video stream is divided into separately solved video slots, and each frame is kept in a buffer for subsequent scheduling. Optical flow from adjacent frames within a video slot is extracted. The standard deviation of the sparse optical flow in each frame is calculated, with the first frame's optical flow defaulting to inf. The number of pixels in each frame whose color depth differs from surrounding pixels by more than a threshold is calculated. Image entropy is calculated. The three features are input into a support vector regression model to predict the F1 score. A list of prediction accuracies for each frame of a single time slot is output. .

[0079] The trained support vector regression model in this embodiment needs to be trained on a certain training set before deployment in the application scenario. This allows the model to predict the detection difficulty of each frame based on some coarse-grained features of the scene. The coarse-grained features used in this embodiment include the number of edge pixels in the image frame, image entropy, and the standard deviation of optical flow between the image frame and the previous image frame. Specifically, drastic image changes are often an important factor affecting image detection accuracy; local object tracking relies on optical flow, which is a two-dimensional vector field in computer vision that describes the pixel motion between two adjacent image frames. It reflects the instantaneous displacement of the object on the image plane. In this embodiment, after collecting the optical flow locally, the standard deviation of optical flow is calculated to measure the degree of fluctuation in the optical flow field. Then, the coarse-grained features corresponding to all image frames are input into the YOLOv7 object detection model, and its F1 value is calculated based on the true labels to obtain the training set.

[0080] Due to features , , The complex relationship between F1 and F1 is not linear and cannot be directly constructed as a simple linear function from data points. Therefore, this embodiment uses a lightweight support vector regression model for fitting and training until the mean squared error (MSE) of the test results reaches a reasonable value. The trained lightweight model is then deployed locally, and at the beginning of each time slot, a feature extractor collects feature information from all frames in the video stream at once. , , Then the model will quickly output a list of prediction accuracies. It is then transmitted to the unloading scheduler for subsequent decision-making modules to evaluate the frame difficulty.

[0081] S103: Based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame, construct the maximum theoretical accuracy expression for each video block, including:

[0082] S103-1: Inference accuracy based on cloud-based models With the characterization of the first In the first time slot Binary parameters indicating whether an image frame is unloaded from the cloud. The product of , to obtain the first In the first time slot Theoretical accuracy of each image frame in the cloud , is represented as: ;

[0083] S103-2: Inference Accuracy Based on Edge Model With the characterization of the first In the first time slot Binary parameters for whether an image frame is unloaded at the edge The product of , to obtain the first In the first time slot Edge theoretical accuracy of each image frame , is represented as: ;

[0084] S103-3: Utilizing the target tracking attenuation mode, based on the first... In the first time slot Theoretical accuracy of each image frame in the cloud Edge theory accuracy Local theoretical accuracy and characterizing the first In the first time slot Binary parameters indicating whether an image frame is unloaded locally. Calculate the first In the first time slot Local theoretical accuracy of each image frame , is represented as: ;

[0085] S103-4: Based on the first Video blocks in each time slot Calculate the maximum theoretical accuracy of the video block by considering the cloud-based theoretical accuracy, edge-based theoretical accuracy, and local theoretical accuracy of each image frame. , is represented as: ;

[0086] in, , This indicates the total number of image frames in the video block; This indicates the preset accuracy attenuation coefficient. This indicates the video frame rate of the video stream to be unloaded.

[0087] Specifically, this embodiment first collects time slots. (Time slot length is) Information on all devices involved in the system, including load data. ,bandwidth Video resolution Video frame rate Processing capacity of edge server deployment model and inference accuracy And the processing capabilities of cloud server deployment models and inference accuracy The three binary parameters defined in this embodiment , and The parameter can be either 1 or 0, which indicates whether a frame is processed on a cloud server, an edge server, or locally, respectively.

[0088] S104: Based on the cloud model processing capability, edge model processing capability, time slot load data, and the length of the preset time slot, construct a time slot load queue; based on the preset time slot delay and preset deadline, construct a time slot delay queue; based on the time slot load queue and the time slot delay queue, construct a Lyapunov function and calculate and obtain the drift term, including steps S104-1 to S104-4.

[0089] S104-1: Constructing a time-slot load queue, including:

[0090] Based on cloud-based model processing capabilities Edge model processing capabilities and the length of the preset time slot The amount of data processed per time slot by the computing system , is represented as: ;

[0091] Based on the system's processing capacity per time slot ,as well as Slot offload data volume Construct a time-slot load queue, represented as:

[0092] ;

[0093] in, Indicates the first System load factor at the end of the time slot Indicates the first System load factor at the end of the time slot This indicates taking the maximum value.

[0094] At the same time, the system must ensure that the load queue is 0 for a long period of time, and the latency cannot exceed the set deadline for each frame. To achieve the maximum theoretical accuracy under constraints, the system applies the Lyapunov optimization framework to address the system stability problem under dynamic networks, constructing the delay constraint as a virtual queue similar to a load.

[0095] S104-2: Constructing a time slot delay queue, including:

[0096] Based on cloud uninstallation ratio The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With cloud bandwidth Calculate cloud latency, expressed as: ;

[0097] Based on edge unloading ratio The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With edge bandwidth The edge delay is calculated and expressed as: ;

[0098] Summing the cloud latency and edge latency, we obtain the latency caused by unloading data within a preset time slot. , is represented as: ;

[0099] Based on the delay and preset deadline of the preset time slot Construct a time slot delay queue, represented as:

[0100] ;

[0101] Among them, frame parameters These are parameters representing the width, height, and data volume of an image frame. Indicates the first The system delay factor at the end of the time slot, Indicates the first The system delay factor at the end of the time slot, This indicates taking the maximum value.

[0102] S104-3: Constructing Lyapunov functions , is represented as:

[0103] ;

[0104] in, Indicates the first State variables of time slots The corresponding Lyapunov function, Indicates the first System load factor at the end of the time slot The square of, Indicates the first System delay coefficient at the end of the time slot The square of.

[0105] S104-4: Calculate and obtain the drift term, expressed as:

[0106] ;

[0107] in, Indicates the first The drift term of each time slot, It represents conditional expectation.

[0108] S105: Based on the drift term and the maximum theoretical accuracy expression, transform to obtain the optimization objective function in draft-plus-penalty form, and take the negative of the optimization objective function as the fitness function;

[0109] The optimization objective function in draft-plus-penalty form is expressed as:

[0110] ;

[0111] in, Indicates the maximum theoretical accuracy. This represents the penalty coefficient.

[0112] Under the proof of Lyapunov optimization theory, the draft-plus-penalty form maintains the same optimal solution as the original problem. Since this problem is a mixed-integer nonlinear programming problem, the optimal solution cannot be found in polynomial time. Therefore, a heuristic genetic algorithm is designed to repeatedly adjust algorithm parameters such as population size. Number of iterations Crossover probability and mutation probability Then, based on the cloud offloading ratio and the edge offloading ratio The variable is the population, and the fitness of the population is the negative of draft-plus-penalty. The near-optimal solution is found through multiple iterations.

[0113] S106: Using the different cloud offloading ratios and edge offloading ratios of each video block as individuals, construct an initial population, update it, and continue until the fitness function converges to obtain the target individuals for each video block.

[0114] Reference Figure 3The diagram shows the flowchart for obtaining the target cloud offloading ratio and the target edge offloading ratio; specifically, it involves collecting data such as the current device load status, network bandwidth information, and the frame rate and resolution of the input video stream; and then using the cloud offloading ratio and the edge offloading ratio as the basis for the calculation. To solve for the target variables, construct an initial population for the genetic algorithm, initialized with random numbers between 0 and 1; calculate the specific frame strategy for each individual in the population; calculate the theoretical accuracy achievable under the frame strategy; calculate the current acceptable load and latency based on video stream data and device data; calculate the Lyapunov-optimized drift-plus-penalty term based on queue data and theoretical accuracy, and use its negative as the population fitness; record the individual with the best fitness in the current population and determine whether the algorithm has converged; if it has not converged or the target number of iterations has been reached, adjust the individual in the population accordingly. Probabilistic crossover produces offspring individuals, and random selection... Probabilistic mutations prevent individuals from getting trapped in local optima; if the algorithm converges, output the best individual in the last generation of the population and its corresponding frame offloading strategy; update the delay load queue and bandwidth data, etc., for use in the next time slot.

[0115] Reference Figure 4 The diagram shows the flowchart for the frame policy generation steps; specifically, for each individual in the population, a list of prediction accuracies is received from the support vector regression model input. and the number of frames within the current video block ;Will Elements and their indices form a new tuple, and the original precision is sorted in descending order; initialize the list. This represents the execution strategy for each frame within a video block. An element with the value "cloud" indicates complex inference is offloaded to the cloud, "edge" indicates normal inference is offloaded to an edge server, and "end" indicates local execution using optical flow information for target tracking. Initially, each element in the list is defined as "end"; depending on the individual... Sort the first few rows in descending order Frame, containing the policy list The element is defined as "cloud"; based on the individual The next step is to unload the frame from the cloud. Frame, containing the policy list The element is defined as "edge"; all remaining unprocessed simple frames are marked as "end"; output frame strategy list. .

[0116] Reference Figure 5 The diagram shown illustrates the calculation flowchart for the theoretical accuracy achieved under the frame strategy; specifically, the list of receiving frame strategies... And based on the theoretical accuracy of cloud and edge models obtained from offline experiments and As input; construct a list of theoretical accuracy , length is And initialize the elements to 0; for the frame policy list The "cloud" element in The index position is assigned a value ; Frame strategy list The "edge" element in The index position is assigned a value ; Frame strategy list The "end" element in The index position is assigned the value of the previous frame's precision multiplied by the attenuation factor. This indicates its theoretical accuracy under optical flow tracing; output a list of theoretical accuracy values. and its mean .

[0117] S107: Based on the target cloud offloading ratio and target edge offloading ratio corresponding to the target individuals in each video block, all image frames in each video block are offloaded to the cloud, edge, or local storage in descending order of prediction accuracy, including:

[0118] Sort all image frames in the video block in descending order of prediction accuracy to obtain the unloading sequence;

[0119] The first in the sequence to be unloaded The frame image is unloaded to the cloud and removed from the unloading sequence, and the unloading sequence is updated.

[0120] The first in the sequence to be unloaded The frame image is unloaded to the edge and removed from the unloading sequence, and the unloading sequence is updated.

[0121] Unload the remaining image frames in the sequence to be unloaded to the local machine;

[0122] in, This indicates the percentage of data uninstalled from the cloud. This indicates the video frame rate of the video stream to be unloaded. This indicates the percentage of data unloaded at the edge.

[0123] This embodiment is based on the target cloud unloading ratio and target edge unloading ratio obtained from the target individual, and the prediction accuracy column given by the accuracy predictor module. The scheduler schedules the frames corresponding to a given time slot in the local buffer pool in descending order of precision, offloading them to the cloud server and edge server for processing, respectively. The simplest remaining frames are kept locally for optical flow tracing. This achieves coarse-grained content exploration frame-level offloading scheduling.

[0124] This invention aims to achieve a stable balance between accuracy, latency, and resource consumption while realizing frame-level scheduling. It applies Lyapunov optimization theory to design an algorithm that dynamically adjusts the proportion of unloaded frames based on system state, and uses a support vector regression model to train an accuracy predictor that assesses the difficulty of image detection, thereby achieving effective frame-level offloading in dynamic and complex network environments and video content. This embodiment predicts the accuracy of each frame within a video block from the input video stream; collects information from all system devices and bandwidth; calculates the target cloud and edge offloading ratios under this system environment as the optimal ratio scheme; and outputs the optimal ratio scheme and the specific scheduling position of each frame under this scheme, thus achieving frame-level scheduling. (Refer to...) Figure 6 The diagram shows the principle of the adaptive video frame offloading method in a cloud-edge-device scenario. Based on the above embodiments, the cloud-edge-device collaborative video analysis technology provided by this invention mainly includes two parts: frame accuracy prediction and dynamic offloading ratio decision.

[0125] Frame accuracy prediction collects coarse-grained features from each frame of the video stream, including the number of edge pixels (pixels whose color depth exceeds a certain threshold with surrounding pixels), image entropy, and the standard deviation of optical flow between the image and the previous frame. These features represent the complexity and variability of the image, respectively. Using these features as input to the accuracy predictor, the output F1 score is used to evaluate the potential complexity of the image in actual detection. Based on this F1 score, the system can determine which image frames require more cloud server resources for inference assistance and which frames can maintain good accuracy locally.

[0126] In the offloading ratio decision-making section, the long-term video analysis is divided into multiple single time slots. Within each time slot, system status is collected, and the optimal ratio is determined. At the beginning of each time slot, the system collects data such as current bandwidth and server load queues. It also locally collects feature information and optical flow for each frame (partly for subsequent local optical flow tracing, and partly for the standard deviation input to the accuracy predictor). Based on the Lyapunov optimization framework, an optimization objective function is defined. After integrating the data, the system heuristically solves for the optimal offloading ratio using a genetic algorithm to achieve the highest population fitness. Subsequently, based on the difficulty coefficient of each frame in the accuracy predictor, the execution strategy for each frame is implemented at the optimal ratio.

[0127] Based on the above embodiments, this invention provides an apparatus for a video frame adaptive offloading method in a cloud-edge-device scenario. The specific apparatus may include:

[0128] The video segmentation module is used to divide the video stream to be unloaded into multiple video blocks according to preset time slots;

[0129] The prediction accuracy acquisition module is used to obtain the prediction accuracy of each image frame based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its previous image frame in the video block.

[0130] The unloading ratio acquisition module is used to construct the maximum theoretical accuracy expression for each video block based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame; construct a time slot load queue based on the processing capacity of the cloud model, the processing capacity of the edge model, the time slot load data, and the length of the preset time slot; construct a time slot delay queue based on the delay and the preset deadline of the preset time slot; construct a Lyapunov function based on the time slot load queue and the time slot delay queue, and calculate and obtain the drift term; based on the drift term and the maximum theoretical accuracy expression, transform and obtain the draft-plus-penalty form of the optimization objective function, and take the negative of the optimization objective function as the fitness function; construct an initial population with different cloud unloading ratios and edge unloading ratios of each video block as individuals, update it, and obtain the target individuals for each video block until the fitness function converges;

[0131] The adaptive unloading module is used to unload all image frames in each video block to the cloud, edge, or local machine in descending order of prediction accuracy, based on the target cloud unloading ratio and target edge unloading ratio corresponding to the target individuals in each video block.

[0132] The video frame adaptive unloading device in the cloud-edge-device scenario of this embodiment is used to implement the aforementioned video frame adaptive unloading method in the cloud-edge-device scenario. Therefore, the specific implementation of the video frame adaptive unloading device in the cloud-edge-device scenario can be found in the embodiment section of the video frame adaptive unloading method in the cloud-edge-device scenario above. For example, the video segmentation module and the prediction accuracy acquisition module are used to implement steps S101 and S102 in the aforementioned video frame adaptive unloading method in the cloud-edge-device scenario, respectively; the unloading ratio acquisition module is used to implement steps S103, S104, S105 and S106 in the aforementioned video frame adaptive unloading method in the cloud-edge-device scenario; and the adaptive unloading module is used to implement step S107 in the aforementioned video frame adaptive unloading method in the cloud-edge-device scenario. Therefore, its specific implementation can be referred to the description of the corresponding embodiments, and will not be repeated here.

[0133] The video frame adaptive offloading method in the cloud-edge-device scenario described in this invention obtains the prediction accuracy of each image frame based on the number of edge pixels, image entropy, and the standard deviation of optical flow between each image frame and its predecessor. It achieves accurate and lightweight quantification of the detection difficulty of each frame based on the coarse-grained features of the image frames, completing frame complexity identification without performing fine-grained convolution operations. This significantly reduces the computational resource consumption of edge devices, avoids the impact of feature extraction on the real-time performance of video analysis, adapts to scenarios with limited edge device resources, and provides a standardized basis for subsequent cloud, edge, or local scheduling allocation based on accuracy, achieving frame-level precise scheduling. This invention applies Lyapunov optimization theory to design an algorithm that can dynamically adjust the proportion of offloaded frames based on system state. It constructs a time-slot load queue by combining the processing capabilities of the cloud and edge models, time-slot load data, and preset time-slot lengths to constrain the hardware processing load of the cloud and edge to within limits. It also constructs a time-slot delay queue by combining time-slot delay and preset deadlines to constrain end-to-end analysis latency to not exceed business requirements. The feasible domain for the offload ratio is defined from the two core dimensions of load and latency, avoiding transmission congestion and system crashes caused by excessive offloading, or wasted cloud computing power caused by insufficient offloading. A Lyapunov function is constructed based on the load queue and virtual queue, and a drift term is calculated to characterize the degree to which the current load and latency deviate from the stable state; the smaller the drift term, the more stable the system. Finally, the maximum theoretical accuracy expression, constructed from the inference accuracy of the cloud model, the inference accuracy of the edge model, and the frame offload location parameters, is transformed into an optimization objective function in the form of a draft-plus-penalty, achieving dual-objective optimization of system stability and analysis accuracy with low load and low latency. Meanwhile, using the cloud offloading ratio and the edge offloading ratio as optimization variables, the optimal solution of the objective function is obtained through a genetic algorithm. This yields the target cloud offloading ratio and the target edge offloading ratio for each video block. This ratio can adapt to the bandwidth, load, and latency status of the current time slot, dynamically determining the number of frames that each video block needs to be offloaded to the cloud or the edge. This achieves accurate matching of the number of offloaded frames in bandwidth fluctuation scenarios, thereby achieving a long-term balance between accuracy, latency, and load, and ensuring the stability of system operation.

[0134] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0135] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A video frame adaptive offloading method in a cloud-edge-device scenario, characterized in that, include: The video stream to be unloaded is divided into multiple video blocks according to preset time slots; The prediction accuracy of each image frame is obtained based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its previous image frame in the video block. Based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame, the maximum theoretical accuracy expression for each video block is constructed. Based on cloud model processing capabilities, edge model processing capabilities, time slot load data, and the preset time slot length, a time slot load queue is constructed, including: based on cloud model processing capabilities Edge model processing capabilities and the length of the preset time slot The amount of data processed per time slot by the computing system , is represented as: Based on the system's processing capacity per time slot ,as well as Slot offload data volume Construct a time-slot load queue, represented as: ;in, Indicates the first System load factor at the end of the time slot Indicates the first System load factor at the end of the time slot This indicates taking the maximum value; Based on the preset time slot delay and preset deadline, a time slot delay queue is constructed, including: based on the cloud offload ratio. The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With cloud bandwidth Calculate cloud latency, expressed as: Based on edge unloading ratio The video frame rate of the video stream to be unloaded With video resolution Length of preset time slot Frame parameters With edge bandwidth The edge delay is calculated and expressed as: Summing the cloud latency and edge latency, we obtain the latency caused by unloading data within a preset time slot. , is represented as: Delay based on preset time slot and preset deadline time Construct a time slot delay queue, represented as: Among them, frame parameters These are parameters representing the width, height, and data volume of an image frame. Indicates the first The system delay factor at the end of the time slot, Indicates the first System delay factor at the end of the time slot; Based on the time slot load queue and the time slot delay queue, a Lyapunov function is constructed, expressed as: ;in, Indicates the first State variables of time slots The corresponding Lyapunov function, Indicates the first System load factor at the end of the time slot The square of, Indicates the first System delay factor at the end of the time slot The square of; And calculate and obtain the drift term, expressed as: ;in, Indicates the first The drift term of each time slot, Expressing conditional expectation; Based on the drift term and the maximum theoretical accuracy expression, the optimization objective function in draft-plus-penalty form is obtained, and the negative of the optimization objective function is taken as the fitness function; Using different cloud offloading and edge offloading ratios of each video block as individuals, an initial population is constructed and updated until the fitness function converges to obtain the target individuals of each video block. Based on the target cloud offloading ratio and target edge offloading ratio corresponding to the target individuals in each video block, all image frames in each video block are offloaded to the cloud, edge, or local in order of prediction accuracy from largest to smallest.

2. The video frame adaptive offloading method in a cloud-edge-device scenario according to claim 1, characterized in that, Based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its predecessor in the video block, the prediction accuracy of each image frame is obtained, including: Get the first video block The gradient of each pixel in each image frame is used to identify edge pixels, and those pixels with gradients greater than a preset gradient threshold are identified as edge pixels. ; Calculate the first video block Image entropy of one image frame , is represented as: ; Calculate the first video block Standard deviation of optical flow between each image frame and its previous image frame , is represented as: ; The video block Image frames , and Input the data into the trained support vector regression model to obtain the first data segment in the video block. Prediction accuracy per image frame; in, Indicates the first video block The grayscale value in each image frame is The normalized probability of pixel occurrence. Indicates the first image frame The size of the optical flow vector per pixel This represents the average value of all optical flow vectors in the image frame. , This represents the total number of pixels in an image frame.

3. The video frame adaptive offloading method in a cloud-edge-device scenario according to claim 1, characterized in that, Based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame, the maximum theoretical accuracy expression for each video block is constructed, including: Cloud-based model inference accuracy With the characterization of the first In the first time slot Binary parameters indicating whether an image frame is unloaded from the cloud. The product of , to obtain the first In the first time slot Theoretical accuracy of each image frame in the cloud , is represented as: ; Inference accuracy based on edge model With the characterization of the first In the first time slot Binary parameters indicating whether an image frame is unloaded at the edge. The product of , to obtain the first In the first time slot Edge theoretical accuracy of each image frame , is represented as: ; Using the target tracking decay mode, based on the first In the first time slot Theoretical accuracy of each image frame in the cloud Edge theory accuracy Local theoretical accuracy and characterizing the first In the first time slot Binary parameters indicating whether an image frame is unloaded locally. Calculate the first In the first time slot Local theoretical accuracy of each image frame , is represented as: ; Based on the Video blocks in each time slot Calculate the maximum theoretical accuracy of the video block by considering the cloud-based theoretical accuracy, edge-based theoretical accuracy, and local theoretical accuracy of each image frame. , is represented as: ; in, , This indicates the total number of image frames in the video block; This indicates the preset accuracy attenuation coefficient. This indicates the video frame rate of the video stream to be unloaded.

4. The video frame adaptive offloading method in a cloud-edge-device scenario according to claim 1, characterized in that, Based on the drift term and the maximum theoretical accuracy expression, the optimization objective function in draft-plus-penalty form is obtained, expressed as: ； in, Indicates the maximum theoretical accuracy. This represents the penalty coefficient.

5. The video frame adaptive offloading method in a cloud-edge-device scenario according to claim 1, characterized in that, Based on the target cloud offloading ratio and target edge offloading ratio corresponding to the target individuals in each video block, all image frames in each video block are offloaded to the cloud, edge, or local storage in descending order of prediction accuracy, including: Sort all image frames in the video block in descending order of prediction accuracy to obtain the unloading sequence; The first in the sequence to be unloaded The frame image is unloaded to the cloud and removed from the unloading sequence, and the unloading sequence is updated. The first in the sequence to be unloaded The frame image is unloaded to the edge and removed from the unloading sequence, and the unloading sequence is updated. Unload the remaining image frames in the sequence to be unloaded to the local machine; in, This indicates the percentage of data uninstalled from the cloud. This indicates the video frame rate of the video stream to be unloaded. This indicates the percentage of data unloaded at the edge.

6. An apparatus based on the video frame adaptive offloading method in a cloud-edge-device scenario as described in any one of claims 1 to 5, characterized in that, include: The video segmentation module is used to divide the video stream to be unloaded into multiple video blocks according to preset time slots; The prediction accuracy acquisition module is used to obtain the prediction accuracy of each image frame based on the number of edge pixels, image entropy, and optical flow standard deviation between each image frame and its previous image frame in the video block. The unloading ratio acquisition module is used to construct the maximum theoretical accuracy expression for each video block based on the inference accuracy of the cloud model, the inference accuracy of the edge model, and the binary parameters representing the unloading position of each image frame; construct a time slot load queue based on the processing capacity of the cloud model, the processing capacity of the edge model, the time slot load data, and the length of the preset time slot; construct a time slot delay queue based on the delay and the preset deadline of the preset time slot; construct a Lyapunov function based on the time slot load queue and the time slot delay queue, and calculate and obtain the drift term; based on the drift term and the maximum theoretical accuracy expression, transform and obtain the draft-plus-penalty form of the optimization objective function, and take the negative of the optimization objective function as the fitness function; construct an initial population with different cloud unloading ratios and edge unloading ratios of each video block as individuals, update it, and obtain the target individuals for each video block until the fitness function converges; The adaptive unloading module is used to unload all image frames in each video block to the cloud, edge, or local machine in descending order of prediction accuracy, based on the target cloud unloading ratio and target edge unloading ratio corresponding to the target individuals in each video block.

Images