Low-redundancy inference method for surveillance video stream based on motion vector

Abstract

The invention discloses a low-redundancy reasoning method for monitoring video streams based on motion vectors, relates to the technical field of image processing, and comprises the following steps: partially decoding an acquired video stream, and extracting motion vector information from partially decoded frame features; inputting the motion vector information and redundant feedback data features into a moving object detector, and when a moving object exists, fully decoding the acquired video stream based on a video decoder to obtain a decoded image; performing region perception division on the decoded image based on a region MOR containing a moving object, reducing spatial dimension redundancy, and obtaining one or more MOR boxes; reasoning the MOR box based on an adaptive reasoning algorithm to obtain a reasoning result, and merging the reasoning result with the decoded image to obtain image information after low-redundancy reasoning; feeding back the time redundancy and space redundancy in the reasoning result to the moving object detector through redundant feedback; the low-redundancy reasoning method improves the overall processing efficiency and speed.

Description

Technical Field

[0001] The invention relates to the field of video processing, and in particular to a low-redundancy reasoning method for monitoring video streams based on motion vectors. Background Art

[0002] Intelligent video analysis is a pattern recognition in artificial intelligence. Through image analysis, various behavior patterns of targets such as people and objects are extracted. In the application, through comparison and contrast, the collected video images are identified to which object and behavior they belong, and the target's action trajectory is labeled to make early warnings and real-time alarms. Intelligent video analysis technology uses machines to implant the "human brain"'s monitoring judgment of video images into the machine through data analysis and feature extraction, forming a "machine brain" to automatically detect and analyze video images, and make alarms or other actions. It uses the powerful data processing capabilities of computers to filter out useless or interfering information in video images, automatically analyze, and extract key useful information from video sources, so that surveillance cameras not only become people's eyes, but also make computers become people's brains. Intelligent video analysis technology is one of the most cutting-edge applications, reflecting the inevitable development trend of future video surveillance systems towards digitalization, intelligence, and diversification.

[0003] With the improvement of video processing technology, the code rate and resolution of video streams are getting larger and larger. In video processing technology, the decoding speed affects the comfort of users watching video streams. However, the existing video analysis system faces a huge computing burden when processing multiple video streams. The traditional decoding and frame-by-frame inference methods have extremely high requirements for computing resources, resulting in low efficiency and speed of existing decoding processing. For example, the Chinese patent with publication number CN110381322B discloses "video stream decoding method, device, terminal device and storage medium", which uses a thread pool corresponding to the GPU to create multiple threads, so as to use the GPU instead of the CPU to complete the multi-channel parallel decoding of video stream data. This method only changes the way of decoding and cannot really reduce the load of video stream decoding. And the Chinese patent with publication number CN113316023B discloses "scene part and interest area processing in video stream transmission", which provides a video stream transmission concept, which allows the transmission of a reduced data stream specific to the scene part while maintaining the consistency between the reduced stream and the original stream, and provides a video stream transmission concept that allows a more efficient way to signal the interest area. This method only provides relevant ideas, but has not been truly implemented. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a low-redundancy reasoning method for monitoring video streams based on motion vectors, which improves the overall processing efficiency and speed.

[0005] The low-redundancy inference method for surveillance video stream based on motion vector proposed in the present invention utilizes the characteristics of continuous video stream to reduce data spatiotemporal redundancy, and comprises the following steps:

[0006] Step 1: partially decode the acquired video stream, and extract motion vector information from the partially decoded frame features;

[0007] Step 2: Input the motion vector information and the data features of the redundant feedback into the moving object detector to detect whether there is a moving object and a moving object region (MOR) in the motion vector information. When there is a moving object, the acquired video stream is fully decoded based on the video decoder to obtain a decoded image, and one or more regions containing moving objects MOR are obtained in the moving object detector;

[0008] Step 3: Based on the MOR of the region containing the moving object, the decoded image is divided into regions based on the position of the moving object to obtain one or more MOR boxes. When there is no moving object, the subsequent decoding and reasoning process is skipped and the result of the previous frame is directly reused;

[0009] Step 4: Infer the MOR box based on the adaptive inference algorithm to obtain the inference result, and merge the inference result with the decoded image to obtain the image information after low redundancy inference;

[0010] Step 5: Feedback the temporal redundancy and spatial redundancy in the inference result to the moving object detector through redundant feedback to optimize the accuracy of the moving object detector in detecting motion vector information and skipping decoding frames.

[0011] Furthermore, in step 1, the process of generating motion vector information is as follows:

[0012] The decoder-based partial decoding is performed on the compressed camera video stream to extract the motion vector encoding information embedded in each compressed frame during the encoding process.

[0013] Furthermore, in step 3, the generation process of the MOR box is as follows:

[0014] Use the minimum enclosing rectangle to enclose all regions MOR as the initial MOR box;

[0015] Divide the minimum enclosing rectangle into partitions num parts, each part corresponds to a MOR box;

[0016] Traverse all regional MORs, evaluate the spatial relationship of each regional MOR with all MOR boxes, and assign the regional MOR to the MOR box with the largest overlap;

[0017] Adjust the size of the MOR box to ensure that the assigned MOR area is covered, evaluate the inferred area of ​​each MOR box, and if the cost increase caused by the adjustment of the MOR box exceeds a preset threshold, reallocate the MOR area with lower cost to the adjacent MOR box;

[0018] All MOR box sizes are normalized to accommodate batch inference;

[0019] Repeat the above steps to select the partition with the smallest total area num The partitioning scheme results in one or more MOR boxes.

[0020] Furthermore, in step 4, the process of generating the inference result is as follows:

[0021] Call the backend model to perform inference on MOR boxes in batches and obtain the inference results of batches of MOR boxes.

[0022] Furthermore, in step 4, the inference result is combined with the decoded image to obtain image information after low-redundancy inference, specifically:

[0023] When there is a moving object, the moving object detector returns the coordinates of the region MOR, and extracts the image and coordinates of each region MOR;

[0024] Extract and adjust the partial bounding box containing the offset from the inference result, and then merge the partial bounding box with the result outside the MOR box;

[0025] The merged bounding boxes are processed by the non-maximum suppression method to eliminate redundant bounding boxes and obtain the final bounding box, which is used as the image information after low-redundancy reasoning.

[0026] Furthermore, in step 5, in optimizing the accuracy of the moving object detector for skipping decoding frames of motion vector information, a model for calculating the decoding probability p of a specific frame is proposed, and its expression is:

[0027] p=α·U+β·R+γ·M

[0028] Among them, U represents the uncertainty measure, R represents redundant feedback, M represents motion vector information feedback, and α, β, and γ are used to balance the contribution of U, R, and M to the overall decoding probability, respectively.

[0029] The advantages of the low-redundancy reasoning method for surveillance video streams based on motion vectors provided by the present invention are: reducing the decoding and inference load of the CPU and GPU, effectively coping with the challenge of concurrent processing; by only analyzing and inferring the motion vectors and the significantly changed region (MOR), these redundancies are greatly reduced, thereby improving the overall processing efficiency and speed; through the strategy of partial decoding and adaptive inference, this embodiment significantly reduces the demand for CPU and GPU resources, reducing energy consumption by about 77.9% and 84.0% respectively. This optimization not only improves the energy efficiency of the system, but also helps to reduce operation and maintenance costs, so that the video analysis system can also operate efficiently in a resource-constrained environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the structural flow of the present invention. DETAILED DESCRIPTION

[0031] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0032] like Figure 1 As shown, the low-redundancy inference method for surveillance video stream based on motion vector proposed by the present invention utilizes the characteristics of continuous video stream to reduce data spatiotemporal redundancy, and includes the following steps:

[0033] Step 1: partially decode the acquired video stream, and extract motion vector information from the partially decoded frame features;

[0034] Partial decoding: After the video stream arrives at the cloud server, instead of reconstructing the complete image frame, partial decoding is performed to extract only the motion vector information. This method greatly reduces the CPU load during concurrent decoding and effectively improves processing speed.

[0035] The process of generating motion vector information is as follows: partially decoding the compressed camera video stream based on the decoder, extracting the motion vector coding information embedded in each compressed frame during the encoding process.

[0036] Step 2: Input the motion vector information and redundant feedback data features into the moving object detector to detect whether there is a moving object in the motion vector information. When there is a moving object, the acquired video stream is fully decoded based on the video decoder to obtain a decoded image. When there is no moving object, the subsequent decoding and inference process is skipped and the result of the previous frame is directly reused;

[0037] Moving object detector: Based on partially decoded motion vector information and continuous redundant feedback data features as input to the moving object detector, the moving object detector can detect the presence and position of moving objects. If no moving object is detected, the current data frame is discarded and the system directly reuses the information of the previous frame; if a moving object is detected, the moving object detector returns the coordinates of the moving object region (MOR), and the video decoder further decodes the complete image frame.

[0038] Step 3: Based on the MOR of the region containing the moving object, the decoded image is divided into regions based on the position of the moving object to reduce the redundancy of the spatial dimension and obtain one or more MOR boxes.

[0039] The generation process of the MOR box is as follows:

[0040] Use the minimum enclosing rectangle to enclose all regions MOR as the initial MOR box;

[0041] Divide the minimum enclosing rectangle into partitions num parts, each part corresponds to a MOR box;

[0042] Traverse all regional MORs, evaluate the spatial relationship of each regional MOR with all MOR boxes, and assign the regional MOR to the MOR box with the largest overlap;

[0043] The size of the MOR box is adjusted to ensure that the assigned MOR area is covered, and the inferred area of ​​each MOR box is evaluated. If the cost increase caused by the adjustment of the MOR box exceeds a preset threshold, the MOR area with a lower cost is reallocated to an adjacent MOR box, thereby obtaining one or more MOR boxes;

[0044] To facilitate batch processing, the sizes of all MOR boxes were standardized;

[0045] Repeat the above steps and finally select the partition with the smallest total area num The partitioning scheme results in one or more MOR boxes.

[0046] Given that there may be multiple regional MORs, the framework adaptively adjusts the region that requires DNN model inference and optimizes based on the location of the regional MOR. Specifically, adjacent MORs are grouped into the same MOR box, and this grouping aims to minimize the area of ​​these boxes, thereby reducing the computational load of inference.

[0047] Step 4: Infer the MOR box based on the adaptive inference algorithm to obtain the inference result, and merge the inference result with the decoded image to obtain the image information after low redundancy inference;

[0048] When there is a moving object, the moving object detector returns the coordinates of the region MOR, extracts the image and coordinates of each region MOR; extracts and adjusts the partial bounding box containing the offset from the inference result, and then merges the partial bounding box with the one outside the MOR box; and considers the inferred area and the specified threshold; processes the merged bounding box by a non-maximum suppression method to eliminate redundant bounding boxes and obtain a final bounding box, and uses the final bounding box as image information after low-redundancy inference.

[0049] Step 5: Feedback the temporal redundancy and spatial redundancy in the inference result to the moving object detector through redundant feedback to optimize the accuracy of the moving object detector in detecting motion vector information and skipping decoding frames.

[0050] When decoding a video stream, the decoding probability p of a specific frame can be calculated by the following formula. When p exceeds the threshold, it is considered that the current frame needs to be decoded for back-end reasoning, thereby optimizing the accuracy of the mobile object detector's detection of motion vector information and skipping decoding frames.

[0051] p=α·U+β·R+γ·M

[0052]

[0053]

[0054]

[0055] Among them, U represents the uncertainty measure, and the uncertainty increases with the increase of the undecoded time, indicating that a higher decoding probability is needed to reduce information loss; last_decode represents the decoding time of the last frame, and max_time_interval represents the maximum allowed skip interval.

[0056] R represents redundant feedback, similarity represents similarity, and similarity_threshold represents the similarity threshold. Similarity is determined by the intersection-over-union (IOU) of the detection results between adjacent frames, which is inversely proportional to the novelty of the information presented in the current frame relative to the previous frames. Therefore, high redundancy reduces the decoding probability. M represents motion vector feedback, mv_nums represents the total number of motion vectors, mv_with_overlap represents the number of overlaps between the motion vector and the bounding box of the moving object in the previous frame, and the motion vector feedback indicates the presence of new objects or moving objects in the frame. Active motion vector feedback increases the possibility of decoding and reflects potential new information.

[0057] The coefficients α, β, and γ are used to balance the contributions of U, R, and M to the overall decoding probability, respectively, to ensure adaptation to different video analysis scenarios.

[0058] Through steps one to five, the core of the low-redundancy inference method for surveillance video streams of this embodiment is to extract motion vector information in the video stream through partial decoding technology, and combine it with moving object detection to determine whether it is necessary to decode the entire frame, effectively reducing time redundancy. At the same time, a region-aware adaptive inference strategy is also set up, which only infers the area containing moving objects, greatly reducing spatial redundancy and computing costs. The combination of these two technologies improves the efficiency of the video analysis system, significantly reduces energy consumption, and ensures the stability of analysis accuracy.

[0059] Compared with the prior art, this embodiment has the following advantages:

[0060] (1) Solving the challenge of real-time concurrent processing: Traditional video analysis systems face huge computational burdens when processing multiple video streams, and traditional decoding and frame-by-frame inference methods have extremely high requirements for computing resources. By introducing partial decoding and intelligent redundant feedback mechanisms, this embodiment significantly reduces the decoding and inference loads of the CPU and GPU, effectively coping with the challenge of concurrent processing.

[0061] (2) Optimizing the processing of temporal and spatial redundancy in video data: There is a large amount of temporal and spatial redundancy in video streams, such as small changes between consecutive frames and objects that move only in local areas. Traditional systems often process each frame in full, resulting in a large amount of unnecessary computational overhead. This embodiment significantly reduces these redundancies by only analyzing and inferring motion vectors and regions with significant changes (MOR), thereby improving overall processing efficiency and speed.

[0062] (3) Maintaining high-precision video analysis quality: While reducing data processing redundancy and speeding up processing, this embodiment can still maintain a high level of analysis accuracy. Through sophisticated moving object detection and region-aware adaptive inference strategies, this embodiment can ensure that only necessary video parts are further processed, thereby maintaining an average accuracy drop of 3.37% while reducing energy consumption and improving speed, which is acceptable in practical applications.

[0063] (4) Achieve efficient energy management: For static scenes without moving objects, this embodiment can identify more than 90% of static scenes, thereby skipping full decoding and inference, significantly reducing CPU and GPU utilization; for moving objects, through partial decoding and adaptive inference strategies, this embodiment significantly reduces the demand for CPU and GPU resources, reducing energy consumption by approximately 77.9% and 84.0%, respectively. This optimization not only improves the energy efficiency of the system, but also helps reduce operation and maintenance costs, allowing the video analysis system to operate efficiently in resource-constrained environments.

[0064] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A low-redundancy inference method for surveillance video stream based on motion vectors, characterized in that: Using the characteristics of continuous video streams to reduce data temporal and spatial redundancy includes the following steps: Step 1: partially decode the acquired video stream, and extract motion vector information from the partially decoded frame features; Step 2: Input the motion vector information and redundant feedback data features into the moving object detector to detect whether there is a moving object and a moving object area in the motion vector information. When there is a moving object, the acquired video stream is fully decoded based on the video decoder to obtain a decoded image. The moving object detector obtains one or more regions MOR containing moving objects. When there is no moving object, the subsequent decoding and reasoning process is skipped and the result of the previous frame is directly reused. Step 3: Based on the MOR of the region containing the moving object, the decoded image is divided into regions based on the position of the moving object, reducing the redundancy of the spatial dimension and obtaining one or more MOR boxes; Step 4: Infer the MOR box based on the adaptive inference algorithm to obtain the inference result, and merge the inference result with the decoded image to obtain the image information after low redundancy inference; Step 5: Feedback the temporal redundancy and spatial redundancy in the inference result to the moving object detector through redundant feedback to optimize the accuracy of the moving object detector in detecting motion vector information and skipping decoding frames.

2. The low-redundancy reasoning method for surveillance video stream based on motion vector according to claim 1 is characterized in that: In step 1, the process of generating motion vector information is as follows: The decoder performs partial decoding on the compressed camera video stream to extract the motion vector encoding information embedded in each compressed frame during the encoding process.

3. The low-redundancy reasoning method for surveillance video stream based on motion vector according to claim 1 is characterized in that: In step 3, the generation process of the MOR box is as follows: Use the minimum enclosing rectangle to enclose all regions MOR as the initial MOR box; Divide the minimum enclosing rectangle into parts, each part corresponds to a MOR box; Traverse all regional MORs, evaluate the spatial relationship of each regional MOR with all MOR boxes, and assign the regional MOR to the MOR box with the largest overlap; Adjust the size of the MOR box to ensure that the assigned regional MOR is covered, evaluate the reasoning area of ​​each MOR box, and if the cost increase caused by the adjustment of the MOR box exceeds a preset threshold, reallocate the regional MOR with lower cost to the adjacent MOR box; All MOR box sizes are normalized to accommodate batch inference; Repeat the above steps and select the one with the smallest total area. The partitioning scheme results in one or more MOR boxes.

4. The low-redundancy reasoning method for surveillance video stream based on motion vector according to claim 1 is characterized in that: In step 4, the process of generating the inference results is as follows: Call the backend model to perform inference on MOR boxes in batches and obtain the inference results of batches of MOR boxes.

5. The low-redundancy reasoning method for surveillance video stream based on motion vector according to claim 1 is characterized in that: In step 4, the inference result is combined with the decoded image to obtain the image information after low redundancy inference, specifically: When there is a moving object, the moving object detector returns the coordinates of the region MOR, and extracts the image and coordinates of each region MOR; Extract and adjust the partial bounding box containing the offset from the inference result, and then merge the partial bounding box with the result outside the MOR box; The merged bounding boxes are processed by the non-maximum suppression method to eliminate redundant bounding boxes and obtain the final bounding box, which is used as the image information after low-redundancy reasoning.

6. The low-redundancy reasoning method for surveillance video stream based on motion vector according to claim 1 is characterized in that: In step 5, in order to optimize the accuracy of the moving object detector in skipping the decoding frame for motion vector information, a method is proposed to calculate the decoding probability of a specific frame. The model is expressed as: in, represents the uncertainty measure, represents redundant feedback, represents motion vector information feedback, , and For balance , , Contribution to the overall decoding probability.

Images