Online video analysis method, system, medium and device

Abstract

The present invention discloses an online video analysis method and system, which includes the following steps: based on the recognition accuracy of the previous time slot and the network status at the current moment, using an online learning algorithm to determine the video frame difference threshold that triggers edge target recognition in the next time slot; before processing each frame, calculating the difference between it and the previous frame, and comparing it with the previously determined threshold to determine whether the current frame is processed on the device side or transmitted to the edge; if processed on the device side, using the optical flow method to calculate the average motion vector and perform simple tracking of the object; if processed on the edge side, using a convolutional neural network to perform target recognition on the entire screen. Using the present invention, the threshold setting can be continuously adjusted according to the intrinsic characteristics of the video and the edge network environment in real applications, maximizing the long-term accuracy of target recognition while ensuring that the real-time performance of video processing is met.

Description

Technical Field

[0001] The present invention relates to video analysis, and in particular to an online video analysis method, system, medium and device for device-side tracking and edge-side identification. Background Art

[0002] In recent years, the number of mobile augmented reality devices and camera-equipped smartphones has rapidly increased. While these devices are widely used, their size limits generally limit their computational capabilities to basic video analysis. However, in many typical video analytics applications, real-time video streams often require rapid processing. For example, real-time augmented reality applications require video processing to quickly classify and locate different objects in the captured image.

[0003] However, detecting and identifying all objects in a real-time video stream online is not an easy task. The challenges lie in three key areas: First, there's the trade-off between the limited resources of the device and the cost of video transmission near the edge; second, the dynamic changes in edge network bandwidth can have uncontrollable impacts on the overall latency of video processing; and finally, the error in object tracking on the device increases over time.

[0004] Specifically, online object detection faces the following multiple challenges: First, performing target detection on the device requires a large amount of computing resources, while transferring the task to the edge requires a lot of transmission time. This requires the entire application to balance between tracking each frame on the device side or transmitting it to the edge for processing. More specifically, because video processing is computationally intensive and often has high resource requirements, deploying it all on the device side can easily lead to long execution delays. For example, complex convolutional neural networks such as YOLO, Faster-RCNN, and Mask-RCNN often require hundreds of milliseconds to detect each frame, which makes it impossible to achieve real-time processing for videos above 24fps. At the same time, offloading the target tracking calculations to the nearby edge will result in the transmission of related frames, and the resulting transmission delay is also not negligible.

[0005] Therefore, to balance on-device computing and video transmission over the edge network, it is possible to combine on-device object tracking with collaborative edge target recognition. However, a major difficulty in designing this solution is that the frequency of uploading images to the edge for processing is limited by the dynamic changes in the edge network environment. For example, in high-quality videos with a resolution higher than 1920x1080, each frame is several megabytes or more in size, and uploading it once will cause a transmission delay of tens of milliseconds. Because peak bandwidth is typically several times higher than minimum bandwidth, the transmission delay of the corresponding video data used for edge inference varies greatly over time.

[0006] For real-time video analytics, the total latency, encompassing tracking on the device, analysis at the edge, and video transmission, must be robustly guaranteed. Otherwise, many frames may need to be queued for processing, leading to network congestion, degraded video processing performance, and a disruptive user experience. Therefore, designing an effective online algorithm to avoid this situation remains a challenging problem.

[0007] Furthermore, the deviations introduced by object tracking on an online device can only be detected after edge correction has been performed, hindering effective online decision-making during the algorithm's execution. Specifically, to reduce the frequency of edge collaborative reasoning, assistance from nearby edges should be sought only when the deviation between the object position obtained by the tracking method and the true position between two consecutive frames is large. However, the object tracking deviation between two consecutive frames is unknown before actual reasoning is performed on the video.

[0008] Existing research is insufficient to address the aforementioned challenges. Previous studies can be categorized into three main categories: one studies on-device processing for edge collaboration to reduce detection complexity and latency, another focuses on how to partition the neural network for video processing between the device and the edge; the remaining studies focus on online edge-assisted device inference within a framework with constrained latency and dynamic edge networks, and theoretically guarantee performance. Summary of the Invention

[0009] Purpose of the invention: The purpose of the present invention is to provide a high-precision, high-efficiency online video analysis method, system, medium and equipment for device-side tracking and edge-side identification.

[0010] Technical solution: The present invention provides an online video analysis method, comprising the following steps:

[0011] (1) Calculate the frame difference threshold using the current bandwidth of the edge network where the device is located and the accuracy and delay information of the target recognition task in the previous time slot;

[0012] (2) Use the optical flow method to calculate the frame difference between the current frame and the previous frame as a measure of the speed of all objects;

[0013] (3) Determine whether the frame difference is greater than the frame difference threshold. If so, upload the image to the edge for processing; if not, track directly on the device.

[0014] (4) If tracking is performed on the device side, the average value of the pixel offset calculated by the optical flow method is used as the motion vector of the object, and the current position of each object is calculated in combination with the position of the object in the previous frame.

[0015] The step (1) is specifically as follows:

[0016] (1.1) Using the network bandwidth information at the beginning of the time slot, the time required to process all frames in the time slot is obtained as:

[0017]

[0018] Among them F t Represents the set of all frames in time slot t, indicating variable I f,t 1 when frame f is uploaded to the edge, 0 when frame f is only processed on the device; Represents the time required to process one frame on the device in time slot t; Represents the time it takes to process a frame on the edge, including transmission latency and convolutional neural network inference latency;

[0019] (1.2) Using the accuracy information of all frames uploaded to the edge in the previous time slot as the optimization target:

[0020]

[0021] in Represents the set of all anchor boxes in the groundtruth of frame f at time t, Represents the accuracy of the target recognition result obtained at the edge. Represents the accuracy of the results obtained on the device side;

[0022] (1.3) Finally, take the maximum accuracy in 1.2 as the optimization goal and the processing time in 1.1 not exceeding the total time of one time slot as the constraint, and solve I f,t The real number solution of , and then use the I of all frames in a time slot to get f,t The sum of the frames to be uploaded is obtained, and the value with the smallest frame difference among the F frames with the largest frame difference in the previous time slot is taken as the solution to the final threshold θ.

[0023] The step (2) is specifically as follows:

[0024] (2.1) For each pixel (x i ,y j ), calculate the motion vector of the pixel

[0025] (2.2) Calculate the Euclidean length of all pixel motion vectors and then find their sum:

[0026]

[0027] Use this value as the frame difference between the current frame and the previous frame.

[0028] The step (3) is specifically as follows:

[0029] (3.1) If the frame difference calculated in step (2.2) is greater than the threshold θ calculated in step (1.3), the edge target recognition module is triggered, and the entire frame is uploaded to the adjacent edge. The convolutional neural network is used to perform the target detection task. At the same time, the result obtained by edge processing of the frame is used as the groundtruth. This is used as the highest accuracy standard to record the accuracy of the frame processing result on the device, which can be expressed as follows:

[0030]

[0031] in, is the set of true positions of each anchor box in frame f in time slot t, and It represents the inferred position set of all anchor boxes of the frame; and Represents the accuracy of the anchor boxes obtained at the edge and device ends respectively, and the accuracy is expressed as the maximum IoU; otherwise, it is directly processed on the device end;

[0032] (3.2) If the value calculated in step (2.2) is less than or equal to the threshold θ calculated in step (1.3), the current frame is processed directly on the device side.

[0033] The step (4) is specifically as follows:

[0034] (4.1) Take the set B of all anchor boxes pred(f) , for each anchor box b=(x b1 ,y b1 ),(x b2 ,y b2 ), take the average value of the motion vectors of all pixels As the direction and distance of movement of the object in the anchor frame;

[0035] (4.2) Anchor frame edge Translate to the new position b'=(x b1 +x b ,y b1 +y b ),(x b2 +x b ,y b2 +yb).

[0036] The present invention also provides an online video analysis system, comprising the following modules:

[0037] The frame difference threshold calculation module is used to calculate the frame difference threshold using the current bandwidth of the edge network where the device is located and the accuracy information and delay information of the target recognition task in the previous time slot;

[0038] The frame difference calculation module is used to calculate the frame difference between the current frame and the previous frame using the optical flow method, which serves as a measure of the speed of all objects;

[0039] A judgment module is used to determine whether the frame difference is greater than the frame difference threshold. If so, the image is uploaded to the edge end for processing; if not, tracking is performed directly on the device end;

[0040] The tracking module is used to calculate the current position of each object based on the average pixel offset calculated by the optical flow method if tracking is performed on the device side.

[0041] A computer storage medium stores a computer program, which implements the above-mentioned online video analysis method when executed by a processor.

[0042] A computer device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the computer program, the online video analysis method is implemented.

[0043] Beneficial effects: Compared with the existing technology, the present invention has the following advantages: the present invention can efficiently combine device-side tracking and edge-side target recognition, while taking into account the dynamic bandwidth in the edge network and the dynamic characteristics of the video. While the algorithm is running, the threshold setting is continuously adjusted according to all previous information, which significantly improves the accuracy and efficiency of real-time video analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A flowchart of the steps of this method is provided;

[0045] Figure 2 Schematic diagram of object tracking based on basic features between two frames. DETAILED DESCRIPTION

[0046] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0047] In video processing applications such as target recognition and real-time target classification, the inference speed of neural networks is slow, and uploading all images to the edge will cause network congestion, making it difficult to meet real-time requirements. However, simply adjusting the frame rate and bit rate of the video cannot guarantee the accuracy of video processing. Therefore, how to combine simple tracking on the device side with using the edge side to perform target recognition under certain conditions to correct the correct position of the object has become an important challenge in real-time video processing. The present invention first proposes a device-side online target recognition method for collaborative edges, and then uses the network and the accuracy information of the previous moment to model the threshold adjustment problem. Based on the problem model, a fast and efficient online learning algorithm is designed.

[0048] like Figure 1 The figure shows an online video analysis method that uses device-side tracking and edge-side recognition. The specific steps include:

[0049] (1) First, at the beginning of each time slot, a suitable threshold is calculated for the current time slot using the accuracy information of the previous time slot and the network bandwidth information at the current moment. The specific steps include:

[0050] (1.1) If the time slot is the first time slot and there is no historical information, then the first four frames are uploaded to the edge for processing, and the average frame difference of the first four frames is used as the initial threshold;

[0051] (1.2) First, use a network speed test tool to obtain the link bandwidth information from the device end to the adjacent edge before the start of each time slot. This gives the first time constraint: in each time slot t, the time to process all frames is:

[0052]

[0053] Among them F t Represents the set of all frames in time slot t, indicating variable I f,t 1 when frame f is uploaded to the edge, 0 when frame f is only processed on the device; Represents the time required to process a frame of image on the device in time slot t. Similarly, Represents the time it takes to process a frame on the edge (including transmission delay and convolutional neural network inference delay).

[0054] (1.3) Then, the accuracy information of all frames uploaded to the edge in the previous time slot is used as the optimization target:

[0055]

[0056] (1.4) Finally, I f,t Relax from integer variables to real variables and transform the problem into a real programming problem;

[0057] Then decouple the problem into a linear programming problem within each time slot, and use the cvxopt library in Python to solve this linear programming problem and quickly solve I f,t The real number solution set of ; then use the I of all frames in a time slot f,t The sum of the frames to be uploaded is obtained, and the value with the smallest frame difference among the F frames with the largest frame difference in the previous time slot is taken as the solution to the final threshold θ.

[0058] (2) For each frame in the current time slot, determine whether it needs to be uploaded to the edge for processing. The specific steps include:

[0059] (2.1) For each pixel (x i ,y j ), using the Lucas-kanade optical flow method, calculate the motion vector of the pixel

[0060] (2.2) Then find the Euclidean length of all pixel motion vectors and sum them up:

[0061]

[0062] (2.3) If the value calculated in step (2.2) is greater than the threshold θ calculated in step (1), the edge target recognition module is triggered, and the entire frame is uploaded to the adjacent edge. The target detection task is processed by the convolutional neural network, and the result obtained by processing the frame at the edge is used as the groundtruth. The accuracy of the frame processing result on the device is recorded with this as the highest accuracy standard, which can be expressed by the following formula:

[0063]

[0064] in, is the set of true positions of each anchor box in frame f in time slot t, and It represents the position set of all anchor boxes of the frame inferred by the method of the present invention. and Represents the accuracy of the anchor box obtained on the edge side and the device side (expressed in terms of maximum IoU); otherwise, it is directly processed on the device side.

[0065] (3) If the processing is performed directly on the device side, the distance the anchor frame is translated from the previous frame to the current frame is determined by the average value of the pixel motion vectors in each anchor frame, and the specific position of the anchor frame in the current frame is obtained. The specific steps include:

[0066] (3.1) Take the set B of all anchor boxes pred(f), for each anchor box b=(x b1 ,y b1 ),(x b2 ,y b2 ), take the average value of the motion vectors of all pixels As the direction and distance of movement of the object in the anchor frame;

[0067] (3.2) Anchor frame edge Translate to the new position b'=(x b1 +x b ,y b1 +y b ),(x b2 +x b ,y b2 +yb).

[0068] (4) Finally, regardless of whether the analysis result of the current frame is obtained on the edge or on the device, the position parameters of the current frame and all anchor frames are returned to the device for rendering and playback. The specific steps include:

[0069] (4.1) Take the set B of all anchor boxes pred(f) , for each anchor box b=(x b1 ,y b1 ),(x b2 ,y b2 ), take the average value of the motion vectors of all pixels As the direction and distance of movement of the object in the anchor frame;

[0070] (4.2) Anchor frame edge Translate to the new position b'=(x b1 +x b ,y b1 +y b ),(x b2 +x b ,y b2 +yb).

[0071] like Figure 2 As shown in the figure, the object tracking process on the device side mainly includes the following steps:

[0072] First, for all anchor frames in the previous frame, take the average of the motion vectors of all pixels in them as the motion vector of the object;

[0073] After calculating the average motion vector, the anchor box is translated along the vector, and the new position obtained is regarded as the prediction of the object position in the current frame.

[0074] The present invention also provides an online video analysis system, which specifically includes the following modules:

[0075] The frame difference threshold calculation module is used to calculate the frame difference threshold using the current bandwidth of the edge network where the device is located and the accuracy information and delay information of the target recognition task in the previous time slot;

[0076] The frame difference calculation module is used to calculate the frame difference between the current frame and the previous frame using the optical flow method, which serves as a measure of the speed of all objects;

[0077] A judgment module is used to determine whether the frame difference is greater than the frame difference threshold. If so, the image is uploaded to the edge end for processing; if not, tracking is performed directly on the device end;

[0078] The tracking module is used to calculate the current position of each object based on the average pixel offset calculated by the optical flow method if tracking is performed on the device side.

[0079] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. An online video analysis method, characterized in that: The method comprises the following steps: (1) Calculate the frame difference threshold using the current bandwidth of the edge network where the device is located and the accuracy and delay information of the target recognition task in the previous time slot; specifically: (1.1) If the time slot is the first time slot and there is no historical information, the first four frames are uploaded to the edge for processing, and the average frame difference of the first four frames is used as the initial threshold; (1.2) If the time slot is not the first time slot, using the network bandwidth information at the beginning of the time slot, the time required to process all frames in the time slot is: Among them F t Represents the set of all frames in time slot t, indicating variable I f,t 1 when frame f is uploaded to the edge, 0 when frame f is only processed on the device; Represents the time required to process one frame on the device in time slot t; Represents the time it takes to process a frame on the edge, including transmission latency and convolutional neural network inference latency; (1.3) Using the accuracy information of all frames uploaded to the edge in the previous time slot as the optimization target: in Represents the set of all anchor boxes in the groundtruth of frame f at time t, Represents the accuracy of the target recognition result obtained at the edge. Represents the accuracy of the results obtained on the device side; (1.4) Finally, taking the maximum accuracy in (1.3) as the optimization goal and the processing time in (1.2) not exceeding the total time of one time slot as the constraint, we can solve I f,t The real number solution of , and then use the I of all frames in a time slot to get f,t The sum of the frames to be uploaded is F, and the value with the smallest frame difference among the F frames with the largest frame difference in the previous time slot is taken as the solution to the final threshold θ; (2) Use the optical flow method to calculate the frame difference between the current frame and the previous frame as a measure of the speed of all objects; (3) Determine whether the frame difference is greater than the frame difference threshold. If so, upload the image to the edge for processing; if not, track directly on the device. (4) If tracking is performed on the device side, the average value of the pixel offset calculated by the optical flow method is used as the motion vector of the object, and the current position of each object is calculated in combination with the position of the object in the previous frame.

2. The method according to claim 1, characterized in that The step (2) is specifically as follows: (2.1) For each pixel (x i ,y j ), calculate the motion vector of the pixel (2.2) Calculate the Euclidean length of all pixel motion vectors and then find their sum: Use this value as the frame difference between the current frame and the previous frame.

3. The method according to claim 1, characterized in that The step (3) is specifically as follows: (3.1) If the frame difference calculated in step (2.2) is greater than the threshold θ calculated in step (1.3), the edge target recognition module is triggered, and the entire frame is uploaded to the adjacent edge. The convolutional neural network is used to perform the target detection task. At the same time, the result obtained by edge processing of the frame is used as the groundtruth. This is used as the highest accuracy standard to record the accuracy of the frame processing result on the device, which can be expressed as follows: in, is the set of true positions of each anchor box in frame f in time slot t, and It represents the inferred position set of all anchor boxes of the frame; and Represents the accuracy of the anchor boxes obtained at the edge and device ends respectively, and the accuracy is expressed as the maximum IoU; otherwise, it is directly processed on the device end; (3.2) If the value calculated in step (2.2) is less than or equal to the threshold θ calculated in step (1.3), the current frame is processed directly on the device side.

4. The method according to claim 1, wherein The step (4) is specifically as follows: (4.1) Take the set B of all anchor boxes pred(f) , for each anchor box b=(x b1 ,y b1 ),(x b2 ,y b2 ), take the average value of the motion vectors of all pixels As the direction and distance of movement of the object in the anchor frame; (4.2) Anchor frame edge Translate to the new position b'=(x b1 +x b ,y b1 +y b ),(x b2 +x b ,y b2 +yb).

5. An online video analysis system, characterized in that: Includes the following modules: The frame difference threshold calculation module is used to calculate the frame difference threshold using the current bandwidth of the edge network where the device is located and the accuracy information and delay information of the target recognition task in the previous time slot; specifically: (1.1) If the time slot is the first time slot and there is no historical information, the first four frames are uploaded to the edge for processing, and the average frame difference of the first four frames is used as the initial threshold; (1.2) If the time slot is not the first time slot, using the network bandwidth information at the beginning of the time slot, the time required to process all frames in the time slot is: Among them F t Represents the set of all frames in time slot t, indicating variable I f,t 1 when frame f is uploaded to the edge, 0 when frame f is only processed on the device; Represents the time required to process one frame on the device in time slot t; Represents the time it takes to process a frame on the edge, including transmission latency and convolutional neural network inference latency; (1.3) Using the accuracy information of all frames uploaded to the edge in the previous time slot as the optimization target: in Represents the set of all anchor boxes in the groundtruth of frame f at time t, Represents the accuracy of the target recognition result obtained at the edge. Represents the accuracy of the results obtained on the device side; (1.4) Finally, taking the maximum accuracy in (1.3) as the optimization goal and the processing time in (1.2) not exceeding the total time of one time slot as the constraint, we can solve I f,t The real number solution of , and then use the I of all frames in a time slot to get f,t The sum of the frames to be uploaded is F, and the value with the smallest frame difference among the F frames with the largest frame difference in the previous time slot is taken as the solution to the final threshold θ; The frame difference calculation module is used to calculate the frame difference between the current frame and the previous frame using the optical flow method, which serves as a measure of the speed of all objects; A judgment module is used to determine whether the frame difference is greater than the frame difference threshold. If so, the image is uploaded to the edge end for processing; if not, tracking is performed directly on the device end; The tracking module is used to calculate the current position of each object based on the average pixel offset calculated by the optical flow method if tracking is performed on the device side.

6. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the online video analysis method according to any one of claims 1 to 4 is implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the online video analysis method according to any one of claims 1 to 4 is implemented.

Images