End-side collaborative real-time video analysis method based on spatio-temporal semantic filtering

Abstract

The application discloses a kind of end side collaborative real-time video analysis methods based on space-time semantic filtering.The method is based on the intelligent vision device and edge server constitute intelligent vision end side collaborative system, then through intelligent vision device to capture video information, through the tracking detection module and region of interest extraction module of carrying configuration.The tracking detection module adaptively determines unloading decision, with tracking algorithm in local processing each video frame, or unload to edge server deduced by object detection model.Region of interest determines the resolution of each unloading frame and detection model configuration, to ensure that analysis result can be returned in time.Tracking detection module and region of interest cooperate with each other, filter repeated space-time semantic information, to maximize processing rate, while ensuring high video analysis accuracy.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of network resource management, and particularly relates to offloading of computing tasks in an edge computing network, and more particularly to an end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering. BACKGROUND

[0002] With the rapid development of the Internet of Things (IoT), the emergence of intelligent visual devices (such as smartphones, autonomous vehicles, and drones) has driven the development of innovative mobile applications such as autonomous driving and virtual reality. These intelligent visual devices perceive the surrounding environment through sensors such as cameras and generate massive amounts of video data that need to be processed in real time. For example, autonomous driving applications use intelligent cameras to accurately detect lanes, cars, and pedestrians to avoid collisions. However, due to the limited computing power of intelligent visual devices, it is very challenging to process computationally intensive and latency-sensitive video analysis tasks locally. Although offloading tasks to powerful cloud servers can alleviate the lack of computing resources, deploying cloud servers far from devices often results in unacceptable latency.

[0003] Edge computing, which deploys powerful edge servers at the network edge, provides ubiquitous, reliable, and fast computing services for devices, making it a promising approach to offloading video analysis tasks to edge servers. Therefore, it is necessary to design a more flexible and efficient end-edge collaborative real-time video analysis method for the following reasons, which can be summarized as follows.

[0004] 1) Due to the limited bandwidth and instability of cellular networks, it is difficult to offload all video frames with large data volumes to edge servers in real time. The 802.11ac design link bandwidth can reach 800Mbps, but the uplink bandwidth of LTE cellular mobile networks is only 50Mbps. Current global mobile network data shows that the actual bandwidth experience of mobile network users is less than 10Mbps.

[0005] 2) The large amount of repeated spatiotemporal semantic information between video frames results in high transmission latency, which seriously affects real-time analysis performance. Since there is a large amount of repeated pictures between adjacent video frames, there is a large amount of repeated semantic information between adjacent video frames. If all information of each video frame is uploaded, a large amount of the same information is repeatedly transmitted, resulting in a huge waste of bandwidth, but without improving the video analysis accuracy.

[0006] 3) Unlike Wi-Fi networks, a large amount of video traffic requires users to pay high traffic fees. At the same time, edge server computing resources are not so easy to obtain, and a large amount of money needs to be spent to purchase from service providers, so reducing the frequency of offloading and the amount of data transmission is crucial for applications that rely on cellular mobile networks for large amounts of data.

[0007] However, it is very challenging to consider all the above facts when designing and utilizing a new efficient edge computing real-time video analysis method for the following reasons:

[0008] a. Real-time video will generate a large amount of traffic in a unit of time, causing visual devices to be unable to transmit all images to edge servers in a timely manner.

[0009] b. The growing number of intelligent visual devices and frequent video analysis task requests will quickly consume the resources of edge servers.

[0010] c. There is a considerable difference in computing power between various intelligent visual devices, causing the performance of the same algorithm to vary greatly, making the design of intelligent algorithms more difficult. SUMMARY

[0011] Invention purpose: For the above-mentioned existing cellular mobile network, the automatic driving, virtual reality, and other computationally intensive and time-sensitive video analysis applications, the present invention provides an end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering.

[0012] Technical solution: An end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering, the method includes constructing an intelligent visual end-edge collaborative system, the system includes intelligent visual devices and edge servers, deploying a tracking detection module and a region of interest extraction module through intelligent visual devices;

[0013] The method takes offloading decision, region of interest extraction, local processing mode, detection model configuration, and processing resolution selection as optimization variables, filters redundant information based on spatiotemporal semantics, and maximizes the frame processing rate and detection accuracy of the edge computing system as the target;

[0014] The method includes the following steps:

[0015] (1) Establish a video analysis task processing model, at the beginning of each time slot, capture video streams through intelligent visual devices, sample to obtain video frames, and then make a decision to allocate whether to offload video frames to edge servers or process them on intelligent visual devices;

[0016] (2) Design an optimizable dynamic data volume calculation model, wherein the calculation of offloaded frame data volume after extracting the region of interest block is:

[0017]

[0018] where τ is the data size of a unit pixel, b(r t，i ) is the number of pixels of the i-th block at resolution r, s t is the number of blocks;

[0019] (4) Determine the complete decision variables of the edge computing system, including frame offloading decision αt, region of interest extraction processing β t , mobile terminal processing mode detection model configuration processing resolution

[0020] (4) Calculate the average latency of the entire system, including the computing latency of the video analysis task on the mobile terminal device pre-detection latency transmission latency computing latency on the edge server total latency of the edge computing system is expressed as follows:

[0021]

[0022] where α t is the decision of whether the mobile device offloads the video frame, β t is the decision of whether the mobile device adopts region of interest extraction;

[0023] (5) Evaluate the detection accuracy of each video frame, including the mobile terminal frame detection accuracy acc(k) and the frame processing accuracy acc(m, r) offloaded to the edge server, which is expressed as follows:

[0024]

[0025] where indicates whether the tracking mode k is adopted, indicates whether the detection model m is used, indicates whether the resolution r is used;

[0026] (6) Based on semantic collaborative filtering, construct the system frame processing rate and accuracy weighted maximization function of the wireless end-edge collaborative network, and the function expression of the optimization problem is as follows:

[0027]

[0028] The constraint conditions are:

[0029] α t , β t ∈{0, 1},

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] where η is a weight parameter of the balance between frame processing rate and detection accuracy, q t is whether the frame detection result is returned in time within the time constraint; if the frame is not processed in time and the frame is discarded; if q t = 0 indicates that the detection accuracy is lower than the detection threshold, q t = 1 indicates that the current frame is successfully processed;

[0038] (8) Training and solving based on reinforcement learning, converting the offloading decision problem in step (6) into a Markov process;

[0039] The offloading decision task state is:

[0040] s t = {h t ,b t ,c t ,p t},

[0041] where h t is the hash similarity, b t is the bandwidth between the edge server and the mobile device, c t is the tracking complexity, and p t is the time for processing in the continuous use tracking mode;

[0042] The offloading decision action space is:

[0043] A t = {Skip, KCF, CSRT, Offload-Full, Offload-ROI}.

[0044] The offloading decision environment reward is:

[0045]

[0046] The offloading decision loss function is:

[0047] L(θ t )=E[(y t -Q(s t ,α t ;θ t )) 2 ]

[0048] (8) Convert the configuration selection problem in step (6) into a context multi-arm bandit problem; the configuration decision is represented as:

[0049]

[0050] The reward estimate used is:

[0051]

[0052] The optimal action is determined using the epsilon-greedy strategy:

[0053]

[0054] The above method is used to solve the video frame resolution and model configuration;

[0055] (9) Obtain frame offloading decision alpha t , region of interest extraction processing beta t , mobile processing mode detection model configuration processing resolution to maximize the weighted sum of system frame processing rate and detection accuracy.

[0056] Further, when the video frame arrives at the intelligent vision device, the intelligent vision device makes a decision on whether to offload the video frame to the edge server or to process it locally through a reinforcement learning control algorithm. The decision factors include video frame redundancy, information quantity, and network bandwidth state.

[0057] The tracking detection module adaptively determines the frame offloading decision alpha t , if alpha t = 0, use the tracking algorithm k and local computing power f device to accelerate processing, otherwise offload it to the edge server for detection using model m and resolution r;

[0058] The region of interest extraction module determines the resolution r and detection model m configuration for each offloaded frame, reducing transmission and inference latency and ensuring that the analysis results are returned in a timely manner.

[0059] The processing mode of the tracking detection module for the video frame includes skipping the current frame, and further includes processing the video frame by the KCF tracker or the CSRT tracker;

[0060] If the video frame is decided by the tracking detection module to be processed on the intelligent vision device, the local processing delay of the current video frame is:

[0061]

[0062] If the video frame is decided by the tracking detection module to be unloaded to the edge server, the frame will be further processed by the region of interest extraction module, and the video frame will be pre-detected by the lightweight deep neural network model;

[0063] In the pre-detection process of the region of interest extraction module, for the network transmission bandwidth that does not meet the data transmission requirement, the video frame is divided into several important video blocks, and the extraction delay is:

[0064]

[0065] For the network transmission bandwidth that meets the data transmission requirement, the video frame will be unloaded in its entirety; however, before unloading, the region of interest extraction module will determine the upload resolution of the video frame and the detection model configuration according to the new information amount of the video frame and the bandwidth state, and the data sending amount is:

[0066]

[0067] Then the video frame is sent to the edge server through the cellular mobile network, and the transmission delay is:

[0068]

[0069] The video frame is inferred on the edge server, and the inference delay is:

[0070]

[0071] After the inference is completed, the detection result is returned to the intelligent vision device, and the detection accuracy of each video frame is evaluated.

[0072] Further, the high-variance normal distribution is used in step (4) to simulate the fluctuation of the cellular mobile network bandwidth, which is mathematically expressed as:

[0073] b t ~ N (p, s).

[0074] According to the wideband state, the transmission delay is calculated as:

[0075]

[0076] The computing amount of the current task is calculated by the task intensity and the data amount, and the time delay of the task processing on different devices is calculated according to the device computing power, wherein the processing time delay of the mobile device end is:

[0077]

[0078] In the formula, u k The computing intensity of the tracking algorithm k, and the time delay of the region of interest extraction is:

[0079]

[0080] In the formula, u roi The computing intensity of the region of interest extraction algorithm, and the time delay of the edge server processing is:

[0081]

[0082] In the formula, u m,r The computing intensity when the resolution r and the model m are used, and the total time delay is finally calculated from these time delays.

[0083] Further, the spatio-temporal redundancy of the video frame is calculated using the hash similarity in step (7), and the similarity degree between adjacent video frames is evaluated, and the specific steps are as follows:

[0084] a) quickly remove high frequency and details by reducing the picture width and height to 112, and only keep the structure brightness;

[0085] b) convert the reduced picture to 64-level gray scale to realize color simplification;

[0086] c) subtract the adjacent two elements of each row of the matrix, and the left element is subtracted from the right element to obtain different difference values;

[0087] d) process the difference values, if it is a positive number or 0, it is recorded as 1; if it is a negative number, it is recorded as 0, and all the results are combined to obtain the hash value;

[0088] e) the similarity of adjacent frames is obtained by calculating the hamming distance of adjacent frames.

[0089] 6. The end-edge collaborative real-time video analysis method based on spatio-temporal semantic filtering according to claim 1, characterized in that the context decision environment of step (8) comprises the following:

[0090] a) high information density high bandwidth context;

[0091] b) low information density high bandwidth context;

[0092] c) high information density low bandwidth context;

[0093] d) low information density low bandwidth context;

[0094] wherein each context environment independently maintains a multi-armed bandit model for adaptive configuration selection, and when the current video frame information density is higher than the average information density, the video frame is identified as a high information density frame;

[0095] The average information density is calculated as follows:

[0096] E i = ξ1n t,i + (1- ξ1) E i ,

[0097] The average bandwidth is calculated as follows:

[0098] E b = ξ2b t + (1- ξ2) E b

[0099] wherein ξ1 is an information density exponential weighted average coefficient, ξ2 represents a bandwidth exponential weighted average coefficient, b t is the bandwidth, and n t,i represents the information density.

[0100] Advantages: Compared with the prior art, the substantial progress and significant effects of the present application are as follows:

[0101] 1) Considering a real-time video analysis system composed of intelligent mobile vision devices, edge servers and cellular mobile networks, a long-term optimization problem is constructed to maximize the frame processing rate and detection accuracy under the constraints of bandwidth and latency. In this system, the novel and efficient DCRL strategy proposed in the present application is adopted, which can optimize the offloading frequency and offloading data volume. In addition, model configuration and resolution selection are also considered as optimization variables.

[0102] 2) The DCRL strategy solves the contradiction between frame processing rate and detection accuracy in video analysis tasks. In particular, the strategy can handle low-bandwidth scenarios with dynamic bandwidth changes in time-varying systems, quantify the spatio-temporal redundancy of video frames using a differential hash perception algorithm, adaptively extract new information content areas, adjust video frame resolution according to the size of the new information introduced by the video frame, and reduce data transmission volume.

[0103] 3) To solve the constructed problem, the present application integrates an offloading decision agent based on DDQN and an adaptive configuration selection agent based on CMAB, which interact with each other to dynamically jointly optimize the decision variables such as offloading mode, video frame resolution, model configuration, etc. according to the changes in video frame information density, bandwidth and tracking state. BRIEF DESCRIPTION OF DRAWINGS

[0104] Figure 1 This is a system model diagram of the method described in this invention;

[0105] Figure 2 This is a diagram of the reinforcement learning solution framework in this invention;

[0106] Figure 3 This is a line graph showing the cumulative rewards under different methods in the example;

[0107] Figure 4 This is a bar chart showing the average detection accuracy under different methods in the example.

[0108] Figure 5 This example demonstrates how to process interest rate bar charts using different methods.

[0109] Figure 6 This is a bar chart showing the average delay under different methods in the example. Detailed Implementation

[0110] To illustrate the technical solutions disclosed in this invention in detail, the invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0111] First, the key problem addressed by the method described in this invention is to propose a real-time video analysis method based on spatiotemporal semantic filtering in an end-to-end collaborative computing system within a wireless cellular mobile network environment. This method reduces redundant information in transmitted video frames, decreases data transmission volume, improves processing speed and efficiency, and maximizes frame processing rate and detection accuracy. The overall model diagram of this method is shown below. Figure 1 As shown.

[0112] The main idea of ​​this invention is to design a real-time video analysis method based on spatiotemporal semantic filtering for edge-end collaboration. This enables a real-time video analysis system based on spatiotemporal semantic filtering for edge-end collaboration, which, with the assistance of other optimization variables, maximizes frame processing rate and detection accuracy. The key element is the designed dynamic application deployment and update strategy: when a task is unloaded to the edge server, if there is no application on the edge server, it will not immediately request the application from the cloud server, but will wait until n tasks have accumulated before requesting the application. When the task queue is empty, the application can still be accessed for a period of time; that is, the edge server will store the application during this time instead of deleting it. If a task arrives during this time, the edge server will continue processing the task until the queue becomes empty again; if no task arrives during this time, the application will be deleted.

[0113] Specifically, an edge-coordinated real-time video analysis method based on spatiotemporal semantic filtering is proposed, the implementation of which includes the following steps:

[0114] Step 1: Establish a video analysis task processing model.

[0115] First, the edge computing network based on intelligent vision equipment, edge server, base station or cloud server is implemented, then based on edge computing, an intelligent vision edge collaborative system of intelligent vision equipment and edge server is constructed, wherein the intelligent vision equipment is used for capturing video data, including a camera or other intelligent sensors for sensing the surrounding environment and other instruments, and relies on the base station to realize cellular mobile network communication and realize data information offloading distribution. Based on the above system and network, a video analysis task processing model is established.

[0116] The video analysis processing model is mainly for video computing task allocation and offloading on the intelligent vision equipment. The intelligent vision equipment has computing and communication capabilities, which can be an intelligent mobile electronic device or other user-side devices that can realize video capture and data processing. For intelligent vision equipment processing, the following processes are included:

[0117] At the beginning of each time slot, a video frame is obtained by sampling the video stream captured by the intelligent vision equipment. Then the video frame is processed by the tracking detection module. The tracking detection module has three processing methods: skipping the current frame, using a KCF tracker, and using a CSRT tracker. Whether the video frame is offloaded is determined by its adaptive decision, which can be considered according to the time bandwidth, computing task amount, etc.

[0118] Further, if the video frame is processed on the intelligent vision equipment, the local processing delay of the current video frame is:

[0119]

[0120] If the video frame is decided to be offloaded to the edge server by the tracking detection module, the frame will be further processed by the region of interest extraction module. The region of interest extraction module first performs pre-detection on the video frame by deploying a lightweight deep neural network model on the intelligent vision equipment. If the bandwidth state is poor and cannot support large data transmission, the region of interest will be extracted, and the video frame will be divided into several important information video blocks. At this time, the extraction delay is:

[0121]

[0122] If the region of interest is not extracted, the video frame will be offloaded in its entirety. However, before offloading, the region of interest extraction module will determine the upload resolution of the video frame and the detection model configuration according to the amount of new information of the video frame and the bandwidth state. At this time, the data transmission amount is:

[0123]

[0124] Then the video frame is sent to the edge server through the cellular mobile network, at this time the transmission delay is:

[0125]

[0126] The video frame is inferred on the edge server, at this time the inference delay is:

[0127]

[0128] After the inference is completed, the detection result is returned to the intelligent vision device, and the detection accuracy of each video frame is evaluated, including the mobile end frame detection accuracy acc(k) and the frame processing accuracy acc(m, r) unloaded to the edge server, and the following calculation expression can be obtained:

[0129]

[0130] The intelligent vision device performs obstacle avoidance, rescue and other tasks according to the detection result. For example, the intelligent vehicle judges whether the detected object will cause an obstacle to the vehicle running according to the detection result, calculates the distance and relative speed between the vehicle and the object and other parameters, and sets an obstacle avoidance strategy according to the threshold value. When encountering an obstacle, a suitable obstacle avoidance strategy is selected, such as turning left or right, slowing down or stopping, etc.

[0131] Step 2, based on the system and model established in step 1, an optimization problem is constructed.

[0132] The application and implementation of the method described in the application include determining the decision variables of the edge computing system, and then solving by algorithm, for which it is necessary to point out that the following five strategy factors are comprehensively considered:

[0133] a) Frame unloading decision alpha t , the value is 0 or 1, representing local processing and unloading processing respectively.

[0134] b) Region of interest extraction processing beta t The response threshold should be greater than or equal to 1 and should be an integer;

[0135] c) Intelligent vision device end processing mode The application waiting duration should be greater than or equal to 0;

[0136] d) Detection model configuration Decide what model each video block or video frame is processed by;

[0137] e) Processing resolution The video frame is processed into this resolution before uploading;

[0138] Further, the average latency of the whole system is calculated, including the video analysis task computing latency on the mobile device Pre-detection latency Transmission latency Computing latency on the edge server Total latency is:

[0139]

[0140] When the frame processing total latency is less than l max , the processing result is returned, and the frame is successfully processed only when the detection accuracy is greater than the threshold. The mathematical representation of the optimization problem is as follows:

[0141]

[0142] The constraint condition is:

[0143] α t , β t ∈{0, 1},

[0144]

[0145]

[0146]

[0147]

[0148]

[0149]

[0150]

[0151] In the formula, η is a weight parameter of the balanced frame processing rate and the detection accuracy. q t is whether the frame detection result is returned in time within the time constraint. If the frame is not processed in time, the frame is discarded. If q t = 0 indicates that the detection accuracy is lower than the detection threshold, q t = 1 indicates that the current frame is successfully processed.

[0152] Step 3, construct a DDQN-based offloading decision agent, and solve the offloading sub-problem of the above optimization problem.

[0153] First, according to the related theory of reinforcement learning, the offloading decision problem is converted into a Markov process; the state of the offloading decision task is:

[0154] s t ={h t ,b,c t ,p t},

[0155] where h t is the hash similarity, b t is the bandwidth between the edge server and the mobile device, c t is the tracking complexity, and p t is the time spent on processing the continuous tracking mode.

[0156] The offloading decision action space is:

[0157] A t ={Skip, KCF, CSRT, Offload-Full, Offload-ROI}.

[0158] The offloading decision environment reward is:

[0159]

[0160] Since the action space is discrete, a deep neural network can be used to analyze the state and action to optimize the Q value. The offloading decision agent observes the state and selects the optimal action to maximize the action value. This process is represented as:

[0161]

[0162] The network parameters are updated using the backpropagation algorithm, where the loss function is:

[0163] L(θ t )=E[(y t -Q(s t ,a t ;θ t )) 2 ].

[0164] However, the DQN-based algorithm can cause large deviations in its model due to overestimation of the Q-target value representing the quality of the policy. To avoid this overestimation, a DDQN-based algorithm is used, which decouples the action selection using DQN and the Q-target evaluation using the target network. In the DDQN algorithm, the value of the Q-target is calculated as:

[0165]

[0166] where θ' t is the target network parameter matrix, and is updated periodically.

[0167] Step 4, build the adaptive configuration selection agent based on CMAB, and solve the configuration sub-problem of the optimization problem in step 3.

[0168] According to the context multi-armed bandit related theory, the configuration selection problem of the optimization problem in the above semantic collaborative filtering is transformed into a context multi-armed bandit problem; comprehensive consideration is given to the following four different context decision environments:

[0169] a) high information density and high bandwidth context;

[0170] b) low information density and high bandwidth context;

[0171] c) high information density and low bandwidth context;

[0172] d) low information density and low bandwidth context;

[0173] Each context environment maintains a multi-armed bandit model independently for adaptive configuration selection. When the information density (bandwidth) of the current video frame is higher than the average information density (average bandwidth), the video frame is identified as a high information density (high bandwidth) frame. The average information density is calculated as follows:

[0174] E i = ξ1n t,i + (1-ξ1)E i ,

[0175] The average bandwidth is calculated as follows:

[0176] E b = ξ2b t + (1-ξ2)E b .

[0177] In order to optimize the solving performance, the joint optimization method is adopted to optimize the video frame resolution and model selection configuration at the same time. The configuration decision is represented as:

[0178]

[0179] The reward estimation is as follows:

[0180]

[0181] The optimal action is determined by the ε-greedy strategy as follows:

[0182]

[0183] The above method is used to solve the video frame resolution and model configuration;

[0184] Step 5, build the optimization solving framework based on DCRL, and solve the optimization problem in step 3.

[0185] Since the tracking and detection module and the region of interest extraction module are highly coupled, and the region of interest extraction module is only used to determine the configuration of each block when the tracking and detection module decides to offload the frame to the ES, this invention proposes a two-layer DDQN-CMAB reinforcement learning (DCRL) framework for jointly training a DDQN-based offloading agent and a CMAB-based configuration selection agent.

[0186] The overall framework diagram for DCRL training is as follows: Figure 2 As shown, the DCRL framework proposed in this invention consists of two layers:

[0187] a) At the upper layer, the tracking and detection module is used as the first-layer controller to observe the hash similarity h from the environment. t Bandwidth b t The tracking complexity is c. t and continuous tracking time p t Then, an unloading decision is made. If the tracking enhancement module decides to process the frame locally, the frame detection result will be obtained using tracking mode k. Otherwise, the frame is sent to the next layer;

[0188] b) In the lower layer, the region of interest extraction module observes the information density n. t,i and bandwidth b t Determine the detection model configuration m and the unloading resolution r for each block.

[0189] In each training step of DCRL, the entire frame of f0 is first unloaded, and the final detection results are initialized for future tracking. Then, in each time slot t, the tracking detection module determines a using DDQN. t If the Skip method is selected, the result of the last test will be used directly as f. t The detection results. If KCF or CSRT mode is used, the tracking and detection module executes a t The corresponding tracking algorithm is used to obtain the detection results. Otherwise, the frame will be processed by the region of interest extraction module, where CMAB is performed to obtain the configuration g for each block. t,i Then, all blocks are unloaded to Elasticsearch. When the results for all blocks are received, the MAB reward R is calculated. e,e To update the estimate Q e,e Simultaneously, TAODM receives the detection results, calculates the reward for the DDQN, and updates its DNN parameter matrix θ. t and the target Q-network parameter matrix θ' t .

[0190] To fully illustrate the edge-end collaborative real-time video analysis method proposed in this invention, its performance will be evaluated using the following three metrics.

[0191] (1) cumulative reward;

[0192] (2) average accuracy;

[0193] (3) processing rate;

[0194] (4) average latency.

[0195] In this embodiment, all the algorithms and simulation experiments implemented by Python are completed on a PC with a 2.30 GHZ CPU and 16G memory, and are trained and tested on the pedestrian tracking dataset. Other comparison methods are as follows:

[0196] Figure 3 The cumulative reward over time under different methods is shown. DCRL is better than other strategies because DCRL filters repeated spatio-temporal semantic information, ensures the timely return of ES results, and at the same time obtains higher accuracy. Figure 4 The average precision (mAP) of different methods is shown. DCRL obtains the highest precision. Figure 5 The frame processing rates of different methods are compared. DCRL has the best processing rate. Specifically, compared with the worst method, DCRL increases the frame processing rate by 66.3%. Compared with Figure 4 and Figure 5 It can be seen that in the case of large transmission delay, the best mAP performance is achieved, and the processing rate is the lowest, while R-H has a higher processing rate and higher mAP due to filtering of repeated spatio-temporal semantic information, which fully demonstrates the effectiveness of ROI.

[0197] Figure 6 The average latency of different methods is compared. D-C has the lowest latency. In the case of tight network environment, the ROI extraction module can effectively reduce the data volume and ensure that the key information is offloaded to the ES for accurate detection. In addition, severe network fluctuations are often temporary. In the case of poor network conditions, the tracking algorithm is used to pass through the network fluctuation period, thereby improving the accuracy.

Claims

1. A real-time video analysis method based on spatiotemporal semantic filtering and edge-end collaboration, characterized in that: This method is based on an intelligent vision edge-to-edge collaborative system, which includes an intelligent vision device and an edge server, and deploys a tracking and detection module and a region of interest extraction module through the intelligent vision device; The method uses unloading decision, region of interest extraction, local processing mode, detection model configuration, and processing resolution selection as optimization variables, and filters redundant information based on spatiotemporal semantics to maximize the frame processing rate and detection accuracy of the edge computing system. The method includes the following steps: (1) Establish a video analysis task processing model. At the beginning of each time slot, capture video streams through intelligent vision devices, then sample them to obtain video frames, and make decisions on whether to offload the video frames to the edge server or the intelligent vision device for processing. (2) Design an optimizable dynamic data volume calculation model. Through this dynamic data calculation model, the calculation of the unloaded frame data volume after extracting the region of interest block can be obtained as follows: ， In the formula, Indicates time slot, It refers to the amount of data per unit pixel. It is the first Block at resolution The number of pixels below, It refers to the number of blocks; (3) Determine the decision variables of the edge computing system, wherein the decision variables include frame offloading decision. Region of Interest Extraction Processing Mobile processing mode Detection model configuration Processing resolution ; (4) Calculate the average latency of the entire system, including the computation latency of the video analysis task on the mobile device. Pre-detection delay Transmission delay Computation latency on edge servers The total latency of this edge computing system It is expressed as follows: , In the formula, This is used to determine whether intelligent vision devices should unload video frames. For intelligent vision devices to decide whether to use region of interest extraction; (5) Evaluate the detection accuracy of each video frame, including the detection accuracy of mobile frames. And frame processing accuracy offloaded to edge servers Two parts, detection accuracy The calculation expression is as follows: , In the formula, Indicates tracking mode Whether to adopt it, Indicates the detection model Whether to use, Represents resolution Whether to use; (6) Based on semantic collaborative filtering, a weighted maximization function for system frame processing rate and accuracy of wireless edge collaborative network is constructed, and the functional expression of the optimization problem is as follows: , The constraints are: , ， ， ， ， ， ， ， In the formula, The weighting parameters are used to balance frame processing rate and detection accuracy. To determine whether the frame detection result is returned in a timely manner within the time constraint; if If a frame is not processed in time, the frame will be discarded; if , This indicates that the detection accuracy is below the detection threshold. This indicates that the current frame has been successfully processed; (7) Training and solving based on reinforcement learning, transforming the unloading decision problem in step (6) into a Markov process; The status of the unloading decision task is: ， In the formula, For hash similarity, For the bandwidth between edge servers and mobile devices, To track complexity, For the duration of continuous use of the tracking mode; The unloading decision action space is: The reward for uninstalling the decision-making environment is: The unloading decision loss function is: (8) Transform the configuration selection problem in step (6) into a contextual multi-armed gambling machine problem; the configuration decision is represented as: . The reward estimate used is: use The greedy strategy determines the optimal action as follows: The video frame resolution and model configuration are solved using steps (7) and (8); (9) Obtain frame offload decisions based on DDQN-based offload decisions and CMAB-based adaptive configuration. Region of Interest Extraction Processing Mobile processing mode Detection model configuration Processing resolution This maximizes the weighted sum of the system's frame processing rate and detection accuracy.

2. The end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering according to claim 1, characterized in that: When a video frame arrives at the intelligent vision device, the intelligent vision device uses a reinforcement learning control algorithm to decide whether the video frame should be offloaded to an edge server or processed locally. The decision-making factors include video frame redundancy, information content, and network bandwidth status. The tracking and detection module adaptively determines the frame offloading decision. ,if Using tracking algorithms and local computing power Accelerate processing, or otherwise offload the model to an edge server. and resolution Conduct testing; The region of interest extraction module determines the resolution of each unloading frame. and detection model Configure settings to reduce transmission and inference latency, ensuring that analysis results can be returned in a timely manner.

3. The end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering according to claim 2, characterized in that: The tracking and detection module handles video frames in several ways, including skipping the current frame and processing video frames using a KCF tracker or a CSRT tracker. If the tracking and detection module determines that the video frame should be processed on the intelligent vision device, the local processing latency of the current video frame is: ； In the formula, For tracking algorithms The calculated intensity, Indicates local computing power; If the tracking and detection module decides to offload the video frame to the edge server, the frame will be further processed by the region of interest extraction module, and the lightweight deep neural network model will perform pre-detection on the video frame. During the pre-detection process of the region of interest extraction module, if the network bandwidth does not meet the data transmission requirements, the video frame is divided into several important video frames. The extraction latency at this time is: In the formula The computational intensity of the region of interest extraction algorithm; If the network bandwidth meets the data transmission requirements, the video frame will be unloaded entirely. However, before unloading, the region of interest extraction module will determine the upload resolution and detection model configuration of the video frame based on the new information content and bandwidth status. At this time, the data transmission volume is: ， The video frames are then sent to the edge server via the cellular network, at which point the transmission latency is: ， Inference is performed on the video frame edge server, and the inference latency is: ， After inference is completed, the detection results are returned to the intelligent vision device to evaluate the detection accuracy of each video frame.

4. The end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering according to claim 1, characterized in that: In step (4), a high-variance normal distribution is used to simulate the bandwidth of a cellular mobile network with large fluctuations, which can be mathematically represented as: The transmission delay calculated based on the broadband status is: ， The computational load of the current task is calculated based on the task intensity and data volume. The processing latency on different devices is calculated based on the device's computing power. The processing latency on the mobile device is as follows: ， In the formula For tracking algorithms The computational intensity and the region of interest extraction latency are: ， In the formula Given the computational intensity of the region of interest extraction algorithm, the edge server processing latency is: In the formula To use resolution and model The computational intensity at each time point is used to calculate the total latency, which is ultimately derived from these latency calculations.

5. The end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering according to claim 1, characterized in that: In step (7), the spatiotemporal redundancy of video frames is calculated using hash similarity, and the similarity between adjacent video frames is evaluated. The specific steps are as follows: a) Quickly remove high frequencies and details by reducing the image's width and height to 112, retaining only structural light and shadow; b) Convert the reduced image to 64 levels of grayscale to simplify the colors; c) Subtract the two adjacent elements in each row of the matrix, subtracting the right element from the left element, to obtain different difference values; d) Process the difference values; if they are positive or 0, remember them as 1. If the result is negative, record it as 0, and combine all the results to obtain the hash value; e) The similarity of adjacent frames is obtained by calculating the Hamming distance between adjacent frames.

6. The end-edge collaborative real-time video analysis method based on spatiotemporal semantic filtering according to claim 1, characterized in that: The contextual decision-making environment mentioned in step (8) includes the following: a) High information density, high bandwidth context; b) Low information density, high bandwidth context; c) High information density, low bandwidth context; d) Low information density, low bandwidth context; Each context environment independently maintains a multi-armed gambling machine model for adaptive configuration selection. When the information density of the current video frame is higher than the average information density, the video frame is identified as a high information density frame. The average information density is calculated as follows: ， The average bandwidth is calculated as follows: In the formula, It is the weighted average coefficient of the information density index. This represents the bandwidth exponent weighted average coefficient. For bandwidth, Indicates information density.

Images