Cloud edge collaboration-based video stream joint inference method and device, medium and equipment

Abstract

The application provides a video stream joint inference method and device based on cloud edge cooperation, a medium and equipment. The method comprises the following steps: adopting a first target detection model of an edge end to perform target detection on a video frame; inputting the video frame into an actor network to obtain a confidence probability; if the confidence probability is greater than a preset probability, sending the video frame and a first target detection result to a cloud end to enable the cloud end to adopt a second target detection model to perform target detection on the video frame; judging whether the cloud end accuracy is higher than the edge end accuracy; if yes, outputting the second target detection result to a user; otherwise, outputting the first target detection result to the user; sending feedback information to the edge end; after receiving the feedback information, calculating an effective offloading ratio, forming state transition information and storing the state transition information into an experience replay pool; selecting state transition information from the experience replay pool and optimizing the actor network. The application can improve the overall accuracy of target detection while ensuring the real-time performance of target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a video stream joint inference method, apparatus, medium, and device based on cloud-edge collaboration. Background Technology

[0002] With the large-scale deployment of smart camera devices in 5G IoT scenarios, the requirements for inference accuracy and end-to-end latency in edge real-time video stream inference analysis for target detection scenarios are becoming increasingly stringent. Using independent edge nodes or cloud nodes for real-time video stream inference analysis is no longer sufficient to meet the demands of today's complex real-time target detection applications. Current solutions for edge real-time video stream inference mostly deploy inference tasks using native Kubernetes objects, lacking operational and governance flexibility. Most solutions that vertically offload video frames to the cloud to improve inference accuracy are too subjective and difficult to adapt to real-time changes in video stream content. Therefore, a new computational paradigm needs to be designed for edge real-time video stream inference analysis.

[0003] Numerous studies have explored combining edge computing with real-time video stream analytics, which has been applied across various fields. In object detection scenarios, many studies also investigate joint inference analysis of real-time video streams. Most of these studies employ two main joint inference paradigms: vertical and horizontal offloading for real-time video frame inference tasks.

[0004] Research on vertical offloading of object detection inference tasks has encountered several challenges. Firstly, the use of different inference model files at the cloud and edge levels leads to significant differences in object detection inference accuracy. Secondly, the research fails to consider whether offloading video frames from the edge to the cloud significantly improves inference accuracy. This makes it difficult to effectively and accurately determine whether a particular video frame needs offloading. Thirdly, the offloading process suffers from subjectivity, randomness, and coarseness, resulting in wasted bandwidth and computing resources at the edge and hindering the overall improvement of real-time video stream object detection inference accuracy. Fourthly, these research solutions lack practicality and application value at resource-constrained edge environments, and cannot guarantee robustness and efficiency when offloading dynamically changing real-time video stream content across different regions. Fifthly, even when the object detection inference accuracy of the inference model files used at the cloud and edge is similar, the aforementioned research still cannot accurately and effectively determine whether all video frame object detection inference tasks at the edge need to be offloaded to the cloud for further inference. Ineffective offloading of a large number of video frame inference tasks would severely waste resources.

[0005] In response to these situations, under the cloud-edge collaborative computing architecture, there is an urgent need to design an excellent collaborative inference vertical offloading algorithm for real-time video stream object detection joint inference tasks to realize the real-time video stream joint inference function, so as to improve the overall accuracy of real-time video stream object detection inference, enhance the continuous stability, real-time performance, availability and reliability of the object detection joint inference service during operation, and make up for the shortcomings and performance limitations of using a single edge terminal or cloud node to manage and control real-time video stream object detection inference applications. Summary of the Invention

[0006] To address at least one of the above technical problems, embodiments of the present invention provide a video stream joint inference method, apparatus, medium, and device based on cloud-edge collaboration.

[0007] According to the first aspect, the cloud-edge collaborative video stream joint inference method provided by the embodiments of the present invention includes:

[0008] The first target detection model at the edge is used to perform target detection on the video frame at the current moment in the video stream acquired by the video acquisition device to obtain the first target detection result;

[0009] The video frame at the current moment is input into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the video frame at the current moment needs to be unloaded to the cloud;

[0010] The confidence probability corresponding to the video frame at the current moment is compared with a preset probability. If the confidence probability corresponding to the video frame at the current moment is greater than the preset probability, the video frame at the current moment and the first target detection result are sent to the cloud. The cloud then uses a second target detection model to perform target detection on the video frame at the current moment to obtain the second target detection result. The cloud target detection accuracy is compared with the first target detection result to determine whether the target detection accuracy is higher than that at the edge. If so, the second target detection result is output to the user as the final detection result. Otherwise, the first target detection result is output to the user as the final detection result. Based on the accuracy determination result, corresponding feedback information is sent to the edge.

[0011] After receiving the feedback information, the current effective offloading ratio is calculated, and the confidence probability corresponding to the video frame at the current moment, the current effective offloading ratio, the image features of the video frame at the current moment, and the image features of the video frame at the next moment are stored as a set of state transition information in the experience replay pool.

[0012] A preset number of state transition information are periodically selected from the experience replay pool, and the actor network is optimized based on the selected state transition information.

[0013] In one embodiment, the confidence probability output by the actor network is an initial confidence probability; correspondingly, before comparing the confidence probability corresponding to the video frame at the current moment with the preset probability, the method further includes: adding random noise to the initial confidence probability output by the actor network to obtain a final confidence probability; wherein the random noise is Gaussian noise or Ornstein-Uhlenbeck noise; correspondingly, comparing the confidence probability corresponding to the video frame at the current moment with the preset probability includes: comparing the final confidence probability corresponding to the video frame at the current moment with the preset probability.

[0014] In one embodiment, the method further includes: inputting the video frame at the current moment and the corresponding initial confidence probability into the commentator network of the edge depth deterministic policy gradient algorithm to obtain the evaluation value of the confidence probability output by the actor network, and optimizing the actor network based on the evaluation value.

[0015] In one embodiment, the actor network includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first flattening layer, a first fully connected layer, and a first output layer connected in sequence; wherein the first convolutional layer and the second convolutional layer are used to perform convolution processing, the first pooling layer and the second pooling layer are used to perform pooling processing, the first flattening layer is used to convert the image features output by the second pooling layer into a one-dimensional vector, and input the one-dimensional vector into the first fully connected layer, and the first output layer outputs an initial confidence probability based on the output features of the first connected layer; and / or, the commentator network The network includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second flattening layer, a second fully connected layer, and a second output layer; wherein, the third and fourth convolutional layers are used to perform convolution processing, the third and fourth pooling layers are used to perform pooling processing, the second flattening layer is used to convert the image features output by the second pooling layer and the initial confidence probability output by the actor network into a one-dimensional vector, and input the one-dimensional vector into the second connected layer, and the second output layer is used to output the comment value corresponding to the initial confidence probability according to the output features of the second connected layer.

[0016] In one embodiment, the method further includes: if the confidence probability corresponding to the video frame at the current moment is less than or equal to the preset probability, then the first target detection result is output to the user as the final detection result.

[0017] In one embodiment, the process of determining whether the target detection accuracy in the cloud is higher than the target detection accuracy at the edge by comparing the second target detection result and the first target detection result includes: if the number of targets in the second target detection result is different from the number of targets in the first target detection result, and / or the target type in the second target detection result is different from the target type in the first target detection result, then the target detection accuracy in the cloud is higher than the target detection accuracy at the edge.

[0018] In one embodiment, the process of the cloud sending corresponding feedback information to the edge terminal based on the accuracy judgment result includes: if the target detection accuracy of the cloud is higher than the target detection accuracy of the edge terminal, then sending feedback information of effective unloading to the edge terminal; otherwise, sending feedback information of invalid unloading to the edge terminal.

[0019] According to a second aspect, the cloud-edge collaborative video stream joint inference device provided in the embodiments of the present invention includes:

[0020] The first detection module is used to perform target detection on the video frame at the current moment in the video stream acquired by the video acquisition device using the first target detection model at the edge, and to obtain the first target detection result.

[0021] The confidence determination module is used to input the video frame at the current moment into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the video frame at the current moment needs to be offloaded to the cloud.

[0022] The probability comparison module compares the confidence probability of the video frame at the current moment with a preset probability. If the confidence probability of the video frame at the current moment is greater than the preset probability, the video frame at the current moment and the first target detection result are sent to the cloud. The cloud then uses a second target detection model to perform target detection on the video frame at the current moment, obtaining the second target detection result. By comparing the second target detection result and the first target detection result, it is determined whether the target detection accuracy of the cloud is higher than the target detection accuracy of the edge. If so, the second target detection result is output as the final detection result to the user; otherwise, the first target detection result is output as the final detection result to the user. Based on the accuracy determination result, corresponding feedback information is sent to the edge.

[0023] The experience collection module is used to calculate the current effective unloading ratio after receiving the feedback information, and store the confidence probability corresponding to the video frame at the current moment, the current effective unloading ratio, the image features of the video frame at the current moment, and the image features of the video frame at the next moment as a set of state transition information in the experience playback pool.

[0024] The network optimization module is used to periodically select a preset number of state transition information from the experience replay pool and optimize the actor network based on the selected state transition information.

[0025] According to a third aspect, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method provided in the first aspect.

[0026] According to a fourth aspect, the computing device provided in the embodiments of the present invention includes a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method provided in the first aspect.

[0027] The present invention provides a cloud-edge collaborative video stream joint inference method, apparatus, medium, and device. At the edge, a first target detection model is used to perform target detection on the current video frame in the video stream acquired by the video acquisition device, obtaining a first target detection result. The current video frame is then input into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the current video frame needs to be offloaded to the cloud. If the confidence probability corresponding to the current video frame is greater than a preset probability, the current video frame and the first target detection result are sent to the cloud. The cloud then uses a second target detection model to perform target detection on the current video frame, obtaining a second target detection result. Finally, the cloud determines the target... If the accuracy of the target detection is higher than that of the target detection at the edge, the second target detection result is output to the user as the final detection result; otherwise, the first target detection result is output to the user as the final detection result. Based on the accuracy judgment, corresponding feedback information is sent to the edge. After receiving the feedback information, the edge calculates the current effective offload ratio and stores the confidence probability of the current video frame, the current effective offload ratio, the image features of the current video frame, and the image features of the next video frame as a set of state transition information in the experience replay pool. A preset number of state transition information is periodically selected from the experience replay pool, and the actor network is optimized based on the selected state transition information. It can be seen that the edge selects a certain number of state transition information from the experience replay pool every so often, and then optimizes the actor network based on the selected state transition information, thereby improving the accuracy of the actor network's output confidence probability. Furthermore, this invention determines whether a video frame needs to be offloaded to the cloud by comparing the confidence probability of the video frame needing to be offloaded to the cloud with a preset probability. This adaptive offload strategy can ensure the robustness of offload performance under dynamically changing real-time video stream content. Target detection at the edge can improve real-time performance, while offloading to the cloud can improve target detection accuracy. Since some video frames are offloaded to the cloud, the waste of computing resources at the edge can be reduced; however, not all video frames are offloaded to the cloud, thus minimizing the need for excessive edge network bandwidth. Therefore, this embodiment of the invention, based on a joint inference mechanism where a low-precision target detection model is provided at the edge and a high-precision target detection inference model is provided in the cloud, achieves complementary inference through cloud-edge collaboration. This effectively ensures the real-time performance of real-time video stream target detection inference while significantly improving the overall accuracy and reliability of real-time video stream target detection inference, maximizing overall inference accuracy without significantly increasing edge network bandwidth consumption or wasting computing resources. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a video stream joint inference method based on cloud-edge collaboration in one embodiment of the present invention.

[0029] Figure 2 This is a structural block diagram of a video stream joint inference device based on cloud-edge collaboration in one embodiment of the present invention. Detailed Implementation

[0030] In a first aspect, embodiments of the present invention provide a video stream joint inference method based on cloud-edge collaboration, the method being executed at the edge, and the method comprising the following steps S110 to S150:

[0031] S110. Use the first target detection model at the edge to perform target detection on the video frame at the current moment in the video stream acquired by the video acquisition device, and obtain the first target detection result;

[0032] For example, the first target detection model is used to perform target detection on the video frame at time t, and the first target detection result is obtained. The detection result contains information such as the number of targets, target type, and target location bounding box.

[0033] S120. Input the video frame at the current moment into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the video frame at the current moment needs to be unloaded to the cloud.

[0034] Among them, the EdgeDepthDeterministicPolicyGradient (DDPG) algorithm is a reinforcement learning algorithm based on the Actor-Critic mechanism. Actor refers to the actor network and Critic refers to the commentator network.

[0035] Among them, the DDPG algorithm can solve both discrete and continuous action problems simultaneously, and it has unique advantages in terms of algorithm training convergence. This algorithm has been used in many studies in recent years. Therefore, this embodiment of the invention takes the DDPG algorithm as the basic guide and, based on the actual application environment of real-time video stream object detection inference tasks, further designs an edge video frame adaptive vertical unloading algorithm, namely the EdgeDDPG algorithm.

[0036] Before inputting video frames into the model, preprocessing is required. Preprocessing operations mainly include converting edge video frames to RGB format and setting the width and height of video frames to a specified size, such as unifying the height and width to 320.

[0037] In one embodiment, the actor network may include a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first flattening layer, a first fully connected layer, and a first output layer connected in sequence; wherein the first convolutional layer and the second convolutional layer are used to perform convolution processing, the first pooling layer and the second pooling layer are used to perform pooling processing, the first flattening layer is used to convert the image features output by the second pooling layer into a one-dimensional vector and input the one-dimensional vector into the first fully connected layer, and the first output layer outputs an initial confidence probability based on the output features of the first connected layer.

[0038] For example, the initial input data of the Actor network is preprocessed edge video frame data of size (320, 320, 3). First, it passes through a first convolutional layer (kernel size 6*3*3, stride 2, padding 0) and a first pooling layer (pooling layer size 2*2, stride 2) to output an 80*80*6 feature image. Then, it passes through a second convolutional layer (kernel size 12*3*3, stride 2, padding 0) and a second pooling layer (pooling layer size 3*3, stride 4) to output a 5*5*12 feature image. Next, a first flattening layer (Flatten layer) is applied to convert the 5*5*12 feature image into a one-dimensional vector of size (1, 300). Then, a fully connected layer (FC layer) with 128 neurons is applied. Finally, an output layer is applied to output the confidence probability of the edge video frames that needs to be unloaded.

[0039] In one embodiment, the commenter network includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second flattening layer, a second fully connected layer, and a second output layer; wherein the third and fourth convolutional layers are used to perform convolution processing, the third and fourth pooling layers are used to perform pooling processing, the second flattening layer is used to convert the image features output by the second pooling layer and the initial confidence probability output by the actor network into a one-dimensional vector, and input the one-dimensional vector into the second connected layer, and the second output layer is used to output the comment value corresponding to the initial confidence probability according to the output features of the second connected layer.

[0040] It is evident that the commentator network and the actor network have similar structures, except that the second flattening layer, in addition to receiving the output features of the fourth pooling layer, also receives the initial confidence probability of the actor network output.

[0041] For example, Critic's initial input data is also preprocessed edge video frame data of size (320, 320, 3). The only difference from the Actor network is that the 5*5*12 feature image is converted into a one-dimensional vector of size (1, 300) after passing through a second flattening layer. At the same time, the number of one-dimensional vectors in the second flattening layer is expanded by one to receive the initial confidence probability of the Actor network output. Then, a fully connected layer with 128 neurons is connected after the second flattening layer, and finally a second output layer with a dimension of 1 that can output the comment value, i.e., the Q value, is connected.

[0042] In one embodiment, the method may further include: inputting the video frame at the current moment and the corresponding initial confidence probability into the commentator network of the edge depth deterministic policy gradient algorithm to obtain the evaluation value of the confidence probability output by the actor network, and optimizing the actor network based on the evaluation value.

[0043] It is evident that after the commenter network obtains the comment values, it can determine the quality of the confidence probability output by the actor network. Furthermore, the parameters of the actor network can be updated based on the comment values ​​to optimize the actor network and improve the accuracy of the confidence probability output by the actor network.

[0044] Wherein, the confidence probability output by the actor network is the initial confidence probability; correspondingly, before executing S130, the method may further include: adding random noise to the initial confidence probability output by the actor network to obtain the final confidence probability; wherein, the random noise is Gaussian noise or Ornstein-Uhlenbeck noise; correspondingly, comparing the confidence probability corresponding to the video frame at the current moment with the preset probability in S130 may include: comparing the final confidence probability corresponding to the video frame at the current moment with the preset probability.

[0045] As can be seen, the EdgeDDPG algorithm adopts an e-greedy strategy to increase random exploration during operation. It adds random noise to the initial confidence probability output by the Actor network with a certain probability and uses it as the confidence probability that the final edge video frame needs to be unloaded. In this embodiment of the invention, Gaussian noise or Ornstein-Uhlenbeck noise can be used as random noise.

[0046] Gaussian noise refers to a type of noise whose probability density function follows a normal distribution.

[0047] Ornstein-Uhlenbeck noise is a stochastic process with temporal correlation, which can enable agents in reinforcement learning to explore effectively in environments with momentum properties.

[0048] S130. Compare the confidence probability corresponding to the video frame at the current moment with a preset probability. If the confidence probability corresponding to the video frame at the current moment is greater than the preset probability, send the video frame at the current moment and the first target detection result to the cloud so that the cloud uses a second target detection model to perform target detection on the video frame at the current moment and obtain the second target detection result. By comparing the second target detection result and the first target detection result, determine whether the target detection accuracy of the cloud is higher than the target detection accuracy of the edge. If so, output the second target detection result as the final detection result to the user; otherwise, output the first target detection result as the final detection result to the user. And send corresponding feedback information to the edge based on the accuracy judgment result.

[0049] The preset probability can be set as needed, for example, to 0.5.

[0050] As can be seen, if the confidence probability corresponding to the video frame at the current moment is greater than the preset probability, the video frame at the current moment and the first target detection result are sent to the cloud. The cloud uses a second target detection model to perform target detection on the video frame at the current moment to obtain the second target detection result; then it is determined whether the target detection accuracy of the cloud is higher than the target detection accuracy of the edge. If the target detection accuracy of the cloud is higher than the target detection accuracy of the edge, then this uninstallation is valid, and the second target detection result is output to the user. If the target detection accuracy of the cloud is not higher than the target detection accuracy of the edge, then this uninstallation is considered invalid, and the first target detection result is output to the user.

[0051] In one embodiment, the process of determining whether the target detection accuracy in the cloud is higher than the target detection accuracy at the edge by comparing the second target detection result and the first target detection result may include: if the number of targets in the second target detection result is different from the number of targets in the first target detection result, and / or the target type in the second target detection result is different from the target type in the first target detection result, then the target detection accuracy in the cloud is higher than the target detection accuracy at the edge.

[0052] Therefore, if the number and / or type of targets in the second target detection result are different from those in the first target detection result, the accuracy of cloud-based target detection is considered to be higher than that of edge-based target detection. If the number and type of targets in the second target detection result are the same as those in the first target detection result, the accuracy of cloud-based target detection is considered to be no higher than that of edge-based target detection.

[0053] In one embodiment, the process of the cloud sending corresponding feedback information to the edge terminal based on the accuracy judgment result may include: if the target detection accuracy of the cloud is higher than the target detection accuracy of the edge terminal, then sending feedback information of effective unloading to the edge terminal; otherwise, sending feedback information of invalid unloading to the edge terminal.

[0054] As can be seen, if the uninstallation is successful, feedback information indicating a successful uninstallation will be sent to the edge. If the uninstallation is unsuccessful, feedback information indicating an unsuccessful uninstallation will be sent to the edge.

[0055] In one embodiment, the method may further include: if the confidence probability corresponding to the video frame at the current moment is less than or equal to the preset probability, then outputting the first target detection result as the final detection result to the user.

[0056] In other words, if the confidence probability of the video frame at the current moment is less than the preset probability, there is no need to unload the data, and the first target detection result is directly output to the user.

[0057] S140. After receiving the feedback information, calculate the current effective offloading ratio, and store the confidence probability corresponding to the video frame at the current moment, the current effective offloading ratio, the image features of the video frame at the current moment, and the image features of the video frame at the next moment as a set of state transition information into the experience replay pool.

[0058] The effective offloading ratio refers to the ratio of the number of times the accuracy improved after the edge video frames were offloaded to the cloud to the total number of times the edge video frames were offloaded to the cloud.

[0059] As can be seen, after receiving feedback information from the cloud, the edge device can determine whether the uninstallation was effective or ineffective, and thus update the effective uninstallation ratio. This generates a set of state transition information, which is then stored in the experience replay pool.

[0060] Each set of state transition information contains four pieces of information: the confidence probability of the video frame at the current moment, the current effective offloading ratio, the image features of the video frame at the current moment, and the image features of the video frame at the next moment. The image features can be a one-dimensional feature vector output from the first flattening layer of the actor network, which can reduce the memory footprint of directly storing edge video frame data in RGB format.

[0061] S150. Periodically select a preset number of state transition information from the experience replay pool, and optimize the actor network based on the selected state transition information.

[0062] That is, the edge network periodically selects a certain number of state transition information from the experience replay pool, and then optimizes the actor network based on the selected state transition information, thereby improving the accuracy of the actor network's output confidence probability. Similarly, the selected state transition information can also be used to optimize the commenter network, thereby improving the accuracy of the commenter network's output comment value.

[0063] As can be seen, the embodiments of the present invention can improve the overall inference accuracy as much as possible without significantly increasing the edge network bandwidth consumption and wasting computing resources.

[0064] This invention determines whether a video frame needs to be offloaded to the cloud by comparing the confidence probability of that the frame needs to be offloaded with a preset probability. This adaptive offloading strategy ensures robustness of offloading performance under dynamically changing real-time video stream content. Target detection at the edge improves real-time performance, while offloading to the cloud improves target detection accuracy. Since some video frames are offloaded to the cloud, the waste of computing resources at the edge is reduced; however, not all video frames are offloaded to the cloud, thus minimizing the need for excessive edge network bandwidth.

[0065] In other words, under the joint reasoning mechanism based on the edge providing a low-precision target detection model and the cloud providing a high-precision target detection reasoning model, the embodiments of the present invention can achieve complementary reasoning through cloud-edge collaboration. While effectively ensuring the real-time performance of real-time video stream target detection reasoning, it can significantly improve the overall accuracy and reliability of real-time video stream target detection reasoning.

[0066] Based on the method provided in this embodiment of the invention, the target detection result received by the user is either the first target detection result obtained by the first target detection model at the edge, or the second target detection result obtained by the second target detection model in the cloud. Through cloud-edge collaboration, the resources of edge computing and cloud computing are fully utilized, improving the flexibility, real-time performance, security, availability, and accuracy limitations of target detection tasks based on real-time video streams when performing inference at the edge.

[0067] This invention utilizes cloud-native technology to achieve real-time video stream target detection joint inference based on cloud-edge collaboration, thereby supporting the automated closed-loop management of real-time video stream target detection joint inference tasks, realizing flexible control and governance of real-time target detection joint inference services, and effectively ensuring the low latency, reliability, accuracy, stability and security requirements of real-time target detection inference tasks during operation.

[0068] This invention presents a cloud-edge collaborative real-time object detection inference solution based on Kubernetes and KubeEdge environments. It addresses the accuracy limitations of current cloud-native environments where inference of real-time video streams using a single edge device or cloud server is limited. Integrating this invention with various object detection scenarios allows data to create greater value.

[0069] This invention addresses the challenge of accurately assessing the performance gap between edge-based and cloud-based object detection models in real-time video streams that dynamically change across different regions. Determining which edge video frames truly need to be unloaded is difficult due to human intervention, leading to uncertainty and unscientific methods. Inappropriate unloading timing can result in ineffective unloading when the accuracy improvement of the cloud-based model is minimal, or edge video frames that should be unloaded are deemed unnecessary. Current edge video frame vertical unloading solutions fail to effectively address these issues. Therefore, this invention designs an adaptive unloading strategy for edge video frames to effectively solve these problems. This strategy automatically identifies edge video frames that, when unloaded to the cloud, can improve the accuracy of object detection inference, thereby enhancing the overall accuracy and reliability of real-time object detection inference tasks as accurately and effectively as possible.

[0070] As can be seen, the embodiments of this invention are essentially an adaptive offloading scheme for edge video frames based on DDPG. Compared to the subjectivity and randomness issues present in some current research schemes when judging the offloading of edge video frames, the embodiments of this invention focus on automatically judging whether each frame of the video stream can effectively improve the accuracy of target detection after offloading, and continuously updating the actor network to achieve self-feedback adjustment, thus avoiding problems such as coarseness, subjectivity, and randomness when offloading edge video frames. At resource-constrained edge devices, the embodiments of this invention can effectively improve the overall accuracy and reliability of real-time video stream target detection inference, reduce the waste of computing power and network bandwidth resources at edge nodes, and adapt to the application requirements of more complex real-time target detection tasks in the future. It can effectively solve the shortcomings and performance limitations of some current research works, and has better design theoretical advantages, practicality, and application value.

[0071] Because real-time video streams inherently possess a certain degree of temporal and spatial locality, the number and location of targets to be detected in multiple consecutive edge video frames may not change significantly. After unloading a certain edge video frame to the cloud, this invention compares the difference in target detection inference accuracy between the cloud model and the edge model. Simultaneously, by leveraging the self-learning and self-feedback mechanisms of reinforcement learning, it can effectively determine whether subsequent consecutive edge video frames similar to the detected targets in that video frame need to be unloaded. This achieves the goal of automatically determining and adjusting the edge video frame unloading strategy, avoiding the need for subjective human intervention in adjusting and controlling the timing and strategy of video frame unloading.

[0072] This invention introduces a reinforcement learning mechanism to automatically adjust the adaptive offloading strategy for edge video frames, thereby identifying the edge video frames that truly need to be offloaded and improving the effective offloading ratio—the ratio of the number of times accuracy improves after an edge video frame is offloaded to the cloud to the total number of times edge video frames are offloaded to the cloud. This maximizes overall inference accuracy without significantly increasing edge network bandwidth consumption or wasting computing resources.

[0073] This invention uses the MOT16 dataset to test and analyze the joint inference performance of the EdgeDDPG edge video frame adaptive offloading algorithm under a high-volume pedestrian target detection task. Several edge video frame offloading algorithms integrated in the Sedna open-source framework are used as benchmark algorithms for performance comparison with EdgeDDPG. The implementation methods of several video frame vertical offloading benchmark algorithms in the Sedna framework are as follows:

[0074] (1) ThresholdFilter algorithm. This algorithm uses the average confidence of all object detection bounding boxes in the edge video frame as the confidence of the object detection result of the edge video frame, and compares it with a confidence threshold to determine whether to perform a vertical unloading operation on the video frame. The default confidence threshold is 0.5.

[0075] (2) CrossEntropyFilter algorithm. This algorithm considers calculating the entropy of the confidence scores of all object detection bounding boxes in the edge video frame as the confidence score of the object detection result of the edge video frame, and compares it with a confidence score threshold to determine the vertical unloading operation of the edge video frame. The default confidence score threshold is 0.5.

[0076] (3) IBTFilter Algorithm. This algorithm calculates the ratio of the number of bounding boxes that meet the confidence threshold to the total number of object detection bounding boxes in the edge video frame, and uses this ratio as the confidence score of the object detection result in the edge video frame. Then, it determines whether the confidence score of the object detection result meets the confidence threshold to decide whether to perform the edge video frame unloading operation.

[0077] (4) RandomFilter algorithm. This algorithm randomly offloads edge video frames with a preset probability value. The default random probability value is 30%.

[0078] The main steps for performing joint inference performance testing on the four algorithms in Sedna are as follows:

[0079] (1) The object detection models YOLOv5s and YOLOv5m were used to perform inference on the second video segment of the MOT16 dataset. After obtaining the pedestrian object detection results, the video frame numbers in which the YOLOv5m model had higher inference accuracy than the YOLOv5s model were counted. These video frames were the ones that actually needed to be unloaded. The second video segment of the MOT16 dataset has a total of 600 frames. The YOLOv5m model had higher inference accuracy than the YOLOv5s model for 453 frames, accounting for 75.50% of the total frames.

[0080] (2) Create a real-time video stream instance VideoStream-1, and fill in the push address of the second video segment of the MOT16 dataset in this instance.

[0081] (3) Create object detection model file instances AIModel-5s and AIModel-5m, and fill in the storage addresses of the Minio objects of Yolov5s and Yolov5m respectively in the instances.

[0082] (4) Create a joint inference task Task-1, specify the edge server to be used as EdgeNode-1 and the cloud server to be used as CloudNode-1; specify the real-time video stream instance and the object detection model file instance used by the edge server EdgeNode-1 as VideoStream-1 and AIModel-5m respectively; specify the object detection model file instance used by CloudNode-1 as AIModel-5m.

[0083] (5) After EdgeNode-1 completes the target detection inference analysis of the second video segment of the MOT16 dataset using the edge video frame vertical unloading algorithm, the video frame number that is actually unloaded to the cloud server CloudNode-1 is counted and compared with the video frame number that actually needs to be unloaded obtained from step (1).

[0084] (6) Repeat the above steps using the third / fifth / ninth video segments of the MOT16 dataset to obtain the video frame number comparison results.

[0085] After testing all the algorithms in Sedna, the average offloading rate, average effective offloading rate, average accuracy improvement rate, average invalid offloading rate, and average number of video frames processed for each algorithm were calculated based on the comparison of the video frame numbers after the tests, using the four videos in the MOT16 dataset. Furthermore, there were significant differences in the accuracy of these algorithms when using the cloud-based Yolov5m model and the edge-based Yolov5s model for object detection inference in the four videos of the MOT16 dataset.

[0086] Among them, the average unloading rate is the ratio of the number of all video frames unloaded by the algorithm to the total number of all video frames; the average effective unloading rate is the number of video frames that the algorithm unloads that truly need to be unloaded; the average accuracy improvement rate is the ratio of the number of video frames that can improve the accuracy of target detection inference after unloading to the total number of all video frames obtained from the real-time video stream; the average invalid unloading rate is the ratio of the number of video frames that do not improve the accuracy of target detection inference after unloading to the total number of unloaded edge video frames; the average number of video frames processed is the number of edge video frames processed per second by the algorithm during model inference and unloading judgment, which reflects the efficiency and operating overhead of the algorithm when performing unloading judgment processing on edge video frames.

[0087] The following is a detailed analysis of the unloading results of several edge video frame vertical unloading algorithms in Sedna:

[0088] (1) The ThresholdFilter algorithm has an average unloading rate of 25%, an average effective unloading rate of 24.69%, an average accuracy improvement rate of 19.67%, an average invalid unloading rate of 21.33%, and an average number of video frames processed of 3.27 frames.

[0089] (2) The average unloading rate of the CrossEntropyFilter algorithm is 100%, the average effective unloading rate is 100%, the average accuracy improvement rate is 75.5%, the average invalid unloading rate is 24.50%, and the average number of video frames processed is 3.09 frames.

[0090] (3) The average unloading rate of the IBTFilter algorithm is 39.67%, the average effective unloading rate is 39.99%, the average accuracy improvement rate is 30.17%, the average invalid unloading rate is 23.95%, and the average number of video frames processed is 3.15 frames.

[0091] (4) The RandomFilter algorithm has an average unloading rate of 28.17%, an average effective unloading rate of 28.04%, an average accuracy improvement rate of 21.17%, an average invalid unloading rate of 25.30%, and an average number of video frames processed of 3.34 frames.

[0092] Next, the EdgeDDPG adaptive offloading algorithm for edge video frames designed in this invention was tested. The testing steps were consistent with those used when testing several algorithms in Sedna. After the EdgeDDPG test was completed, the specific offloading analysis results are shown in Table 1.

[0093] Table 1. Results of the unloading test of the EdgeDDPG algorithm.

[0094]

[0095] A comprehensive analysis of the unloading test results for these four video segments in the MOT16 dataset reveals that the EdgeDDPG algorithm achieved an average unloading rate of 70.48%, an average effective unloading rate of 71.31%, an average accuracy improvement rate of 58.26%, an average invalid unloading rate of 15%, and an average of 2.97 video frames processed.

[0096] In addition, by repeatedly modifying the Actor and Critic network architecture design in EdgeDDPG and using Gaussian noise and Ornstein-Uhlenbeck noise in the network respectively, the offloading analysis results of all edge video frames after retesting did not have a very significant advantage compared with the above results. Moreover, the indicators of the results after offloading are all better than the several algorithms in Sedna, and the overall offloading performance can be maintained at a good level.

[0097] Comparative testing and analysis show that the EdgeDDPG algorithm has a higher average offloading rate, average effective offloading rate, and average accuracy improvement rate compared to the ThresholdFilter, IBTFilter, and RandomFilter algorithms. EdgeDDPG has the lowest average ineffective offloading rate compared to these three algorithms, significantly improving inference accuracy without significantly increasing the ineffective offloading rate, effectively avoiding the waste of edge network bandwidth and computing resources caused by a large number of invalid video frames. Regarding the average number of video frames processed, the EdgeDDPG algorithm can complete offloading judgment processing and model inference for 2.97 video frames per second, which is only slightly reduced compared to the 3.27, 3.15, and 3.34 frames per second of the other algorithms. Therefore, the EdgeDDPG algorithm does not significantly increase processing time overhead during offloading judgment, remaining within an ideal and tolerable range. Although EdgeDDPG slightly increases processing time overhead during operation, it has a more significant average accuracy improvement rate, achieving ideal performance and efficiency while significantly improving object detection inference accuracy compared to other algorithms. In terms of identifying edge video frames that truly need to be unloaded and unloading efficiency, the EdgeDDPG algorithm clearly outperforms other algorithms overall.

[0098] Although the CrossEntryFilter algorithm boasts the highest average effective offloading rate and average accuracy improvement, reaching 100%, it offloads all video frames used in the test for the four video segments. In practical applications, if there is no difference in inference accuracy between the edge inference model and the cloud inference model, this algorithm is clearly undesirable and would severely waste resources. This embodiment of the invention suggests that this situation is likely due to an inappropriate selection of the confidence threshold parameter in the algorithm. Therefore, subjectively determining the relevant parameters in Sedna's built-in video frame offloading algorithm to ensure offloading performance under dynamically changing real-time video stream content presents certain challenges and lacks reliability and stability.

[0099] The EdgeDDPG algorithm in this embodiment of the invention demonstrates significant advantages in offloading performance and operational efficiency compared to several algorithms in Senda, particularly in practical applications. It effectively avoids issues of subjectivity, randomness, and coarseness when determining whether edge video frames need offloading, thereby improving the overall accuracy of real-time video stream target detection inference and reducing resource waste to some extent. Furthermore, it ensures robustness in offloading performance when dealing with video stream content that changes in real-time across different geographical locations.

[0100] Besides the edge video frame offloading algorithm in the Sedna framework, several other video frame offloading algorithms in mainstream literature research are also quite representative, such as VideoEdge, DeepDecision, and EdgeOnly. Performance tests and comparisons will follow.

[0101] The VideoEdge algorithm requires offline measurement of the resource configuration relationship between the resource usage requirements and inference accuracy of the inference model files used in video frame inference tasks at various video frame resolutions. With fixed node CPU and network bandwidth resources, it maximizes the average accuracy of all object detection inference tasks on the edge node by automatically selecting the optimal resource configuration relationship for different inference tasks. Unlike EdgeDDPG, which provides a high-precision object detection model in the cloud and a low-precision model at the edge in this embodiment of the invention, VideoEdge uses the same inference model at both the edge and the cloud for each inference task. When the edge node resources meet the requirements, VideoEdge will always choose the resource configuration relationship that maximizes the average inference accuracy of all object detection inference tasks. It determines whether object detection inference for different tasks can be performed at the edge or in the cloud based on the CPU and network bandwidth resource usage at the edge, and can select the optimal resource configuration parameters and execution location for each object detection inference task to maximize the average inference accuracy. This differs from the architecture of the present invention, which determines whether to offload edge video frames to the cloud model by comparing the inference accuracy difference between two models in order to maximize the improvement of inference accuracy.

[0102] The DeepDecision algorithm shares a similar architecture with this invention, providing a low-precision inference model on the local device and a high-precision inference model on the remote server. However, it doesn't effectively and accurately determine whether to unload a particular edge video frame based on the difference in inference accuracy between the two models. Instead, it dynamically adjusts the video bitrate and frame resolution to provide the user with optimal model parameter selection to maximize inference accuracy, taking into account dynamic changes in network bandwidth and battery consumption. However, this approach still leads to unnecessary resource waste if the inference accuracy improvement of the model on the remote server is not significant compared to the model on the local device. This scheme fails to effectively and accurately measure the difference in inference accuracy between the two models and cannot determine whether unloading edge video frames will necessarily improve object detection accuracy during inference.

[0103] Both VideoEdge and DeepDecision, without pre-defined resource constraints, will select the edge video frame offloading scheme that yields the highest accuracy for object detection inference among available options. Therefore, when comparing them with EdgeDDPG in this embodiment of the invention, both can offload the edge video frame inference task to the cloud, using a high-precision inference model in the cloud to maximize the accuracy of real-time video stream inference. Simultaneously, the comparison with EdgeDDPG will focus on metrics such as average network bandwidth consumption at the edge, time cost to complete all video frame inference tasks, average offloading rate, average effective offloading rate, average accuracy improvement rate, and average ineffective offloading rate.

[0104] When testing VideoEdge and DeepDecision, after offloading the pedestrian detection inference tasks for the second, third, fifth, and ninth video segments of the MOT16 dataset to a high-precision inference model in the cloud, the average edge network bandwidth consumption during this period was approximately 513.6 KB / s, and the time to complete the inference task for all video frames (i.e., the time from capture of all video frames to the final object detection inference result being uploaded to the cloud) was approximately 4419.31 seconds. The average offloading rate was 100%, the average effective offloading rate was 100%, the average accuracy improvement rate was 75.50%, and the average invalid offloading rate was 24.50%. After testing the EdgeDDPG algorithm, the average edge network bandwidth consumption during this period was approximately 289.5 KB / s, and the time to complete the inference task for all video frames was approximately 3625.98 seconds. The average offloading rate was 70.48%, the average effective offloading rate was 71.31%, the average accuracy improvement rate was 58.26%, and the average invalid offloading rate was 15%.

[0105] In addition, this embodiment of the invention also tests the EdgeOnly algorithm, which performs inference only at the edge, to better illustrate the advantages of EdgeDDPG in improving the overall inference accuracy of real-time object detection when there is a significant performance gap between cloud and edge models. After the test, the average edge network bandwidth consumption during this period was 0KB / s, and the inference time cost to complete the inference task for all video frames was approximately 2481.50 seconds. The average offload rate was 0%, the average effective offload rate was 0%, the average accuracy improvement rate was 0%, and the average invalid offload rate was 0%.

[0106] Comparative analysis reveals that, compared to VideoEdge and DeepDecision's strategy of offloading all edge video frame inference tasks to the cloud, EdgeDDPG consumes less average network bandwidth at the edge, has a shorter time overhead to complete all video frame inference tasks, and significantly improves the accuracy of real-time video stream object detection inference. However, if the performance difference in inference accuracy between the cloud and edge models is not as significant as in the test environment, and edge network bandwidth and computing resources are limited, offloading all video frame inference tasks to the cloud, while certainly improving object detection inference accuracy, will severely impact the real-time performance of object detection tasks, potentially failing to meet user application latency requirements, significantly reducing the reliability of object detection inference services, and causing unnecessary resource waste. Therefore, when accurate real-time object detection inference results are not required, the EdgeDDPG algorithm, when running on resource-constrained edge nodes, possesses certain theoretical design advantages, practicality, and application value compared to strategies that offload all edge video frame inference tasks to the cloud without preliminary screening, demonstrating superior overall offloading performance and operational efficiency. In real-world scenarios, EdgeDDPG or other offloading strategies can be selected based on the available resources at the edge and the actual requirements of the application.

[0107] Compared to the EdgeOnly algorithm, which performs inference only at the edge, EdgeDDPG achieves a higher average inference accuracy improvement rate and a lower average invalid offloading rate, without significantly wasting edge network bandwidth and computing resources. While there is a significant performance gap in inference accuracy between cloud and edge models, if the latency increase from offloading edge video frames to the cloud is within a tolerable range, EdgeDDPG can substantially improve the accuracy of object detection inference, compensating for the shortcomings and limitations of edge-only inference and effectively ensuring the reliability and accuracy of real-time object detection inference applications.

[0108] Based on the above experimental tests and analyses, when there is a significant performance gap in inference accuracy between cloud and edge models, the EdgeDDPG algorithm in this invention demonstrates a more significant overall advantage over benchmark solutions such as Sedna, VideoEdge, and DeepDecision in improving the overall inference accuracy of real-time video stream object detection tasks on resource-constrained edge devices. EdgeDDPG also exhibits good performance in terms of runtime overhead and execution efficiency, effectively ensuring the effectiveness and performance robustness of offloading edge video frames when dealing with dynamically changing real-time video stream content across different regions, thus enhancing its practicality and application value.

[0109] To better illustrate the performance advantages and theoretical value of the EdgeDDPG algorithm in vertically unloading edge video frames when there is a significant gap in object detection inference accuracy between cloud and edge models, this embodiment of the invention uses the second video segment from the MOT16 dataset as an example to introduce the dynamic changes in unloading rate, effective unloading rate, and ineffective unloading rate of EdgeDDPG, Sedna, VideoEdge, and DeepDecision when performing unloading judgment processing on all video frames in this video. Based on the comparison of the dynamic changes of relevant test indicators, EdgeDDPG has significant advantages over Sedna, VideoEdge, and DeepDecision in terms of unloading rate, effective unloading rate, and ineffective unloading rate, demonstrating superior unloading performance and operational stability. Considering the gap in object detection inference accuracy between cloud and edge models, for this video, EdgeDDPG's unloading rate during runtime is closest to the expected unloading rate. The number of video frames that should have needed unloading to truly improve object detection inference accuracy in this video is 453, and the expected unloading rate is 75.50%. EdgeDDPG not only boasts a higher offloading rate but also a higher effective offloading rate and the lowest invalid offloading rate. While ensuring the real-time performance of real-time object detection inference tasks, it effectively and accurately determines that offloaded edge video frames, once offloaded to the cloud, can improve object detection inference accuracy and avoid unnecessary resource waste. Furthermore, after conducting the same offloading process on the third, fifth, and ninth video segments of the MOT16 dataset with EdgeDDPG, Sedna, VideoEdge, and DeepDecision, a comprehensive comparison of offloading rate, effective offloading rate, and invalid offloading rate revealed that EdgeDDPG's overall offloading performance remains superior. This demonstrates that EdgeDDPG effectively ensures robustness and reliability in improving object detection inference accuracy when facing different real-time video stream content. Solutions such as Sedna, VideoEdge, and DeepDecision exhibit unstable offloading performance on these videos. They are inferior to EdgeDDPG in all metrics and suffer from significant coarsening and randomness when offloading video frames. They cannot effectively and accurately determine whether a particular edge video frame truly needs to be offloaded to the cloud, making it difficult to improve the overall accuracy of real-time video stream inference and ensure the robustness and reliability of offloading performance.

[0110] The performance comparison and analysis of EdgeDDPG with benchmark solutions such as Sedna, VideoEdge, and DeepDecision showed that the inference model files used by the cloud and edge nodes were Yolov5m and Yolov5s, respectively. A significant difference in accuracy was observed when inferring four video segments from the MOT16 dataset. EdgeDDPG demonstrated a higher average effective offloading rate, a higher average accuracy improvement rate, and a lower average ineffective offloading rate, resulting in a more significant overall offloading performance. If there is no difference in accuracy between the inference model files used by the cloud and edge nodes when inferring videos from the MOT16 dataset, then all video frames from the edge should not be offloaded to the cloud. Offloading would be ineffective, failing to improve accuracy and severely wasting edge network bandwidth and computing resources.

[0111] This invention also tests and analyzes the accuracy of several algorithms—EdgeDDPG, Sedna, VideoEdge, and DeepDecision—in determining whether edge video frames need to be unloaded by setting up an experimental environment where there is no difference in object detection inference accuracy between cloud and edge models. During testing, both cloud and edge nodes used Yolov5s inference model files, and the dataset used consisted of the second, third, fifth, and ninth video segments from the MOT16 dataset. When there is no difference in inference accuracy performance between cloud and edge models, the algorithm achieves the best unloading performance if it determines that all video frames do not need to be unloaded (i.e., an average unloading rate of 0%), effectively avoiding waste of network bandwidth and computing resources at edge nodes.

[0112] When testing four algorithms in Sedna, the average offloading rate was 25% for ThresholdFilter, 100% for CrossEntropyFilter, 39.67% for IBTFilter, and 28.17% for RandomFilter. All four algorithms offloaded a large number of video frames, and failed to effectively and accurately determine that all video frames in the four videos in the MOT16 dataset did not need to be offloaded to the cloud.

[0113] When testing VideoEdge and DeepDecision, because they unload all edge video frames to the inference model file of the cloud node, the average unloading rate of the algorithm is 100%, and it is still unable to effectively and accurately determine that all edge video frames do not need to be unloaded.

[0114] When testing the EdgeDDPG algorithm in this embodiment of the invention, the average unloading rate of the algorithm was 11.65%. The number of video frames unloaded to the cloud during operation is small, which can effectively and accurately determine that most edge video frames do not need to be unloaded.

[0115] After the above tests were completed, the EdgeDDPG algorithm was compared with four algorithms in Sedna, VideoEdge and DeepDecision in terms of the average unloading rate of the four videos.

[0116] A comprehensive analysis of the comparison results of the above algorithms in terms of average offload rate shows that the EdgeDDPG algorithm has the lowest average offload rate. When there is no difference in inference accuracy between the cloud and edge models, it can effectively avoid invalid offloading of edge video frames and effectively reduce the waste of network bandwidth and computing resources at edge nodes.

[0117] To better illustrate the performance advantages and theoretical value of the EdgeDDPG algorithm in avoiding vertical offloading of edge video frames when there is no difference in object detection inference accuracy between cloud and edge models, this invention still uses the second video segment from the MOT16 dataset as an example to introduce the dynamic changes in the offloading rate of EdgeDDPG, Sedna, VideoEdge, and DeepDecision when performing offloading judgment processing on all video frames in this video. Comparing the dynamic changes in the offloading rate, EdgeDDPG has the lowest offloading rate compared to Sedna, VideoEdge, and DeepDecision. EdgeDDPG's dynamic offloading rate remains consistently low, effectively avoiding invalid offloading of edge video frames and significantly reducing the waste of edge network bandwidth and computing resources.

[0118] The EdgeDDPG algorithm consistently delivers better overall offloading performance regardless of whether there is no significant difference in inference accuracy between cloud and edge models. The EdgeDDPG algorithm not only effectively improves the overall accuracy and reliability of real-time video stream object detection inference, but also effectively determines whether the offloading of edge video frames is ineffective, significantly reducing the waste of edge network bandwidth and computing resources. In practical applications, its self-feedback learning mechanism demonstrates stronger performance robustness and stability during offloading, resulting in higher application and practical value.

[0119] Since real-time object detection joint inference tasks may require the use of a large number of edge nodes during deployment and operation, and different real-time video streams may be allocated to these nodes, the inference performance of the object detection model file will vary depending on the dynamically changing real-time video stream content in different regions when unloading it. If the determination of whether a certain edge video frame needs to be unloaded is still done subjectively, randomly, and roughly, it will be difficult to guarantee the performance and efficiency of unloading large-scale real-time video stream inference tasks, and it will be impossible to comprehensively and effectively improve the overall accuracy of large-scale real-time video stream object detection inference tasks. The EdgeDDPG algorithm in this embodiment of the invention can dynamically adapt to different real-time video stream content, and can automatically and self-feedback adjust the unloading strategy for different real-time video stream content. This can significantly improve the overall accuracy and reliability of large-scale real-time object detection inference tasks, and provide users with reliable vertical unloading support for real-time object detection inference tasks.

[0120] Understandably, real-time video stream object detection joint inference tasks require the use of a large number of edge nodes during deployment and operation, and different real-time video streams need to be allocated to these nodes. When offloading these dynamically changing real-time video stream contents from different regions, the EdgeDDPG object detection collaborative inference vertical offloading algorithm designed in this embodiment of the invention, after comparison with mainstream algorithms, proves that it can dynamically adapt to different real-time video stream contents. It has the ability to automatically and self-feedback adjust the offloading strategy for different real-time video stream contents, which can significantly improve the overall accuracy and reliability of large-scale real-time video stream object detection inference tasks, providing users with reliable vertical offloading support for real-time video stream object detection inference tasks.

[0121] The EdgeDDPG joint inference algorithm automatically measures the performance gap in inference accuracy between cloud and edge models and makes offloading decisions. It significantly improves the overall accuracy of real-time object detection inference tasks while ensuring real-time performance, effectively reducing the waste of edge network bandwidth and computing resources. Compared to several widely used mainstream edge video frame vertical offloading algorithms, it offers better offloading performance, effectively identifying truly needed offloading methods with higher effective offloading rates and accuracy improvements, without significantly increasing ineffective offloading rates or runtime overhead. Its overall offloading performance is superior on publicly available datasets, avoiding the subjectivity and uncertainty inherent in manually measuring performance differences between inference models.

[0122] More specifically, compared to Sedna, a cloud-edge collaborative framework that has garnered widespread attention in recent years, its initial integration of cloud-edge collaborative joint inference capabilities, while widely applied, still presents certain limitations and shortcomings. The embodiments of this invention offer significant advantages over Sedna in terms of joint inference functionality. Several edge video frame offloading algorithms provided by Sedna struggle to effectively determine the confidence threshold parameters when faced with different video stream content, leading to uncertainties, subjectivity, and randomness in offloading performance, severely limiting its ability to improve the overall accuracy of real-time video stream target detection inference. The method provided in these embodiments can automatically adjust the offloading strategy based on the real-time changing video stream content to maximize the identification of video frames that truly improve target detection accuracy after offloading, significantly increasing the effective offloading rate.

[0123] The deep integration of cloud-native technologies and artificial intelligence (AI) is an inevitable trend. Designing a joint inference scheme for real-time video stream object detection at the edge can adapt to future AI application scenarios with higher demands on computing power and architecture. This invention analyzes the pain points of current real-time video stream object detection inference in cloud-native environments and proposes a joint inference algorithm scheme using cloud-native technologies to optimize the management cost and efficiency of real-time video stream inference, simplify the deployment process of joint inference service applications, promote the automation level of application execution, achieve faster updates and deployment of joint inference services, accelerate the innovative application of AI in various industries, and enable AI technology to play a greater social and economic role.

[0124] Secondly, embodiments of the present invention provide a video stream joint inference device based on cloud-edge collaboration, see [link to relevant documentation]. Figure 2 The device 100 includes:

[0125] The first detection module 110 is used to perform target detection on the video frame at the current moment in the video stream acquired by the video acquisition device using the first target detection model at the edge end, and to obtain the first target detection result.

[0126] The confidence determination module 120 is used to input the video frame at the current moment into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the video frame at the current moment needs to be offloaded to the cloud.

[0127] The probability comparison module 130 is used to compare the confidence probability corresponding to the video frame at the current moment with a preset probability. If the confidence probability corresponding to the video frame at the current moment is greater than the preset probability, the video frame at the current moment and the first target detection result are sent to the cloud so that the cloud uses a second target detection model to perform target detection on the video frame at the current moment to obtain the second target detection result. By comparing the second target detection result and the first target detection result, it is determined whether the target detection accuracy of the cloud is higher than the target detection accuracy of the edge. If so, the second target detection result is output to the user as the final detection result; otherwise, the first target detection result is output to the user as the final detection result. And according to the accuracy judgment result, corresponding feedback information is sent to the edge.

[0128] The experience collection module 140 is used to calculate the current effective unloading ratio after receiving the feedback information, and store the confidence probability corresponding to the video frame at the current moment, the current effective unloading ratio, the image features of the video frame at the current moment and the image features of the video frame at the next moment as a set of state transition information into the experience playback pool.

[0129] The network optimization module 150 is used to periodically select a preset number of state transition information from the experience replay pool and optimize the actor network based on the selected state transition information.

[0130] In one embodiment, the confidence probability output by the actor network is the initial confidence probability; correspondingly, before comparing the confidence probability corresponding to the video frame at the current moment with the preset probability, the confidence determination module is further configured to: add random noise to the initial confidence probability output by the actor network to obtain the final confidence probability; wherein, the random noise is Gaussian noise or Ornstein-Uhlenbeck noise; correspondingly, the comparison of the confidence probability corresponding to the video frame at the current moment with the preset probability in the confidence determination module includes: comparing the final confidence probability corresponding to the video frame at the current moment with the preset probability.

[0131] In one embodiment, the apparatus further includes:

[0132] The network optimization module is used to input the video frame at the current moment and the corresponding initial confidence probability into the commentator network of the edge depth deterministic policy gradient algorithm, obtain the evaluation value of the confidence probability output by the actor network, and optimize the actor network based on the evaluation value.

[0133] In one embodiment, the actor network includes a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first flattening layer, a first fully connected layer, and a first output layer connected in sequence; wherein the first convolutional layer and the second convolutional layer are used to perform convolution processing, the first pooling layer and the second pooling layer are used to perform pooling processing, the first flattening layer is used to convert the image features output by the second pooling layer into a one-dimensional vector, and input the one-dimensional vector into the first fully connected layer, and the first output layer outputs an initial confidence probability based on the output features of the first connected layer; and / or, the commentator network The network includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second flattening layer, a second fully connected layer, and a second output layer; wherein, the third and fourth convolutional layers are used to perform convolution processing, the third and fourth pooling layers are used to perform pooling processing, the second flattening layer is used to convert the image features output by the second pooling layer and the initial confidence probability output by the actor network into a one-dimensional vector, and input the one-dimensional vector into the second connected layer, and the second output layer is used to output the comment value corresponding to the initial confidence probability according to the output features of the second connected layer.

[0134] In one embodiment, the apparatus further includes:

[0135] The first output module is configured to output the first target detection result as the final detection result to the user if the confidence probability corresponding to the video frame at the current moment is less than or equal to the preset probability.

[0136] In one embodiment, the process of determining whether the target detection accuracy in the cloud is higher than the target detection accuracy at the edge by comparing the second target detection result and the first target detection result includes: if the number of targets in the second target detection result is different from the number of targets in the first target detection result, and / or the target type in the second target detection result is different from the target type in the first target detection result, then the target detection accuracy in the cloud is higher than the target detection accuracy at the edge.

[0137] In one embodiment, the process of the cloud sending corresponding feedback information to the edge terminal based on the accuracy judgment result includes: if the target detection accuracy of the cloud is higher than the target detection accuracy of the edge terminal, then sending feedback information of effective unloading to the edge terminal; otherwise, sending feedback information of invalid unloading to the edge terminal.

[0138] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents of the apparatus provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0139] Thirdly, embodiments of the present invention provide a computer-readable medium storing computer instructions, which, when executed by a processor, cause the processor to perform the method provided in the first aspect.

[0140] Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the embodiments described above is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0141] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0142] Examples of storage media used to provide program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0143] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0144] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion module connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion module execute some and all of the actual operations, thereby realizing the function of any of the above embodiments.

[0145] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the contents in the computer-readable medium provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0146] Fourthly, one embodiment of this specification provides a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method of any embodiment of the specification.

[0147] It is understood that explanations, specific implementation methods, beneficial effects, examples, etc. of the computing device provided in the embodiments of the present invention can be found in the corresponding parts of the method provided in the first aspect, and will not be repeated here.

[0148] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, widgets, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0149] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video stream joint inference method based on cloud-edge collaboration, characterized in that, include: The first target detection model at the edge is used to perform target detection on the video frame at the current moment in the video stream acquired by the video acquisition device to obtain the first target detection result; The video frame at the current moment is input into the actor network of the edge depth deterministic policy gradient algorithm to obtain the confidence probability that the video frame at the current moment needs to be unloaded to the cloud; The confidence probability corresponding to the video frame at the current moment is compared with a preset probability. If the confidence probability corresponding to the video frame at the current moment is greater than the preset probability, the video frame at the current moment and the first target detection result are sent to the cloud so that the cloud uses a second target detection model to perform target detection on the video frame at the current moment to obtain the second target detection result. The cloud target detection accuracy is compared with the first target detection result to determine whether the target detection accuracy at the cloud is higher than the target detection accuracy at the edge. If so, the detection result of the second target will be output to the user as the final detection result; Otherwise, the detection result of the first target will be output to the user as the final detection result; And based on the accuracy judgment result, send corresponding feedback information to the edge end; After receiving the feedback information, the current effective offloading ratio is calculated, and the confidence probability corresponding to the video frame at the current moment, the current effective offloading ratio, the image features of the video frame at the current moment, and the image features of the video frame at the next moment are stored as a set of state transition information in the experience replay pool. A preset number of state transition information are periodically selected from the experience replay pool, and the actor network is optimized based on the selected state transition information; The process of determining whether the target detection accuracy in the cloud is higher than that at the edge by comparing the second target detection result and the first target detection result includes: if the number of targets in the second target detection result is different from the number of targets in the first target detection result, and / or the target type in the second target detection result is different from the target type in the first target detection result, then the target detection accuracy in the cloud is higher than that at the edge.

2. The method according to claim 1, characterized in that, The confidence probability output by the actor network is the initial confidence probability; Correspondingly, before comparing the confidence probability corresponding to the video frame at the current moment with the preset probability, the method further includes: adding random noise to the initial confidence probability output by the actor network to obtain the final confidence probability; wherein, the random noise is Gaussian noise or Ornstein-Uhlenbeck noise; Correspondingly, comparing the confidence probability corresponding to the video frame at the current moment with the preset probability includes: comparing the final confidence probability corresponding to the video frame at the current moment with the preset probability.

3. The method according to claim 2, characterized in that, Also includes: The video frame at the current moment and the corresponding initial confidence probability are input into the commentator network of the edge depth deterministic policy gradient algorithm to obtain the evaluation value of the confidence probability output by the actor network, and the actor network is optimized based on the evaluation value.

4. The method according to claim 2, characterized in that, The actor network comprises a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, a first flattening layer, a first fully connected layer, and a first output layer connected in sequence; wherein, the first convolutional layer and the second convolutional layer are used to perform convolution processing, the first pooling layer and the second pooling layer are used to perform pooling processing, the first flattening layer is used to convert the image features output by the second pooling layer into a one-dimensional vector, and input the one-dimensional vector into the first fully connected layer, and the first output layer outputs an initial confidence probability based on the output features of the first connected layer; And / or, the commenter network includes: a third convolutional layer, a third pooling layer, a fourth convolutional layer, a fourth pooling layer, a second flattening layer, a second fully connected layer, and a second output layer; wherein, the third and fourth convolutional layers are used to perform convolution processing, the third and fourth pooling layers are used to perform pooling processing, the second flattening layer is used to convert the image features output by the second pooling layer and the initial confidence probability output by the actor network into a one-dimensional vector, and input the one-dimensional vector into the second connected layer, and the second output layer is used to output the comment value corresponding to the initial confidence probability according to the output features of the second connected layer.

5. The method according to claim 1, characterized in that, Also includes: If the confidence probability corresponding to the video frame at the current moment is less than or equal to the preset probability, then the first target detection result is output to the user as the final detection result.

6. The method according to claim 1, characterized in that, The process of the cloud sending corresponding feedback information to the edge terminal based on the accuracy judgment result includes: if the target detection accuracy of the cloud is higher than the target detection accuracy of the edge terminal, then sending feedback information of effective unloading to the edge terminal; otherwise, sending feedback information of invalid unloading to the edge terminal.

Images