A Fast Video Review Method and System Based on Temporal Prediction

Abstract

The present invention discloses a fast video review method and system based on temporal prediction, mainly related to the field of computer vision technology. The method includes the following steps: constructing a temporal state prediction model, globally scanning the initial frame of the video stream, and calling a target detection algorithm to identify and locate the target; extracting the state information of the target in the initial frame to form an initial state vector, and initializing the temporal state prediction model; using the initialized temporal state prediction model to perform iterative loop processing on each frame after the initial frame to complete video review. The beneficial effect of the present invention is that by predicting the possible positions and states of the target in subsequent frames, it focuses computing resources on scanning and identifying the predicted region of interest (ROI), rather than performing global scanning, thereby greatly saving computing resources and improving the review efficiency.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a fast video review method and system based on temporal prediction. Background Art

[0002] With the popularization of applications such as video surveillance, content review, and behavior analysis, it has become an important requirement to efficiently and accurately process and analyze a large amount of video data. Traditional video review or analysis methods usually involve global scanning and object detection for each frame of the video. Although this method can obtain comprehensive information, when dealing with high-resolution and high-frame-rate video streams, it consumes a large amount of computing resources (such as CPU and GPU processing time) and bandwidth resources, resulting in slow processing speed and high cost, and it is difficult to meet the requirements of real-time or large-scale processing.

[0003] To improve efficiency, researchers have proposed some methods based on object tracking. However, existing methods may not fully utilize the motion state information of the object to actively predict its future position, or are limited to specific tracking algorithms.

[0004] Therefore, there is an urgent need for a technical solution that can significantly reduce the computational complexity and improve the video review processing speed. Summary of the Invention

[0005] The purpose of the present invention is to provide a fast video review method and system based on temporal prediction, which scans and identifies by concentrating computing resources on the predicted region of interest (ROI) by predicting the possible positions and states of the object in subsequent frames, rather than performing global scanning, thereby greatly saving computing resources and improving the review efficiency.

[0006] To achieve the above object, the present invention is realized through the following technical solutions:

[0007] On the one hand, the present invention provides a fast video review method based on temporal prediction, including the following steps:

[0008] Step S1: Construct a temporal state prediction model, perform a global scan on the initial frame of the video stream, and call an object detection algorithm to identify and locate the object;

[0009] Step S2: Extract the state information of the object in the initial frame to form an initial state vector , and initialize the temporal state prediction model;

[0010] Step S3: Use the initialized temporal state prediction model to perform iterative loop processing on each frame after the initial frame to complete video review.

[0011] Preferably, in the step S2, the state information of the target in the initial frame includes, but is not limited to, the position information of the target of interest , the size information , and the motion parameters estimated through multi-frame data, where the motion parameters include, but are not limited to, the velocity information , the acceleration information .

[0012] Preferably, the step S3 includes the following steps:

[0013] S31: Define each frame after the initial frame as the current frame , and predict the current frame through the sequential state prediction model based on the corrected state of the previous frame ;

[0014] S32: Generate the region of interest in the current frame according to the position information and size information of the predicted state , where the size of the region of interest is the product of the predicted size and the magnification factor , where ;

[0015] S33: Perform target analysis within the region of interest, where the target analysis includes, but is not limited to: precise positioning, feature extraction, and classification recognition;

[0016] S34: If the target is detected within the region of interest, obtain the measurement state of the target, and use to correct the sequential state prediction model, and input the posterior state as the input for the next frame prediction.

[0017] Preferably, the generation of the region of interest satisfies the following conditions:

[0018] The width of the region of interest is: , and the height is: , where .

[0019] Preferably, the target analysis includes, but is not limited to:

[0020] Precisely positioning the target within the region of interest using a deep learning-based detector;

[0021] Extracting the texture, color, or shape features of the target to enhance the tracking robustness;

[0022] Identifying the category attributes of the target through a classification model.

[0023] ​​Preferably, for model correction, the Kalman gain is used to calculate the weighted fusion of the predicted state and the measured state, or the weight parameters of the deep learning model are updated through the backpropagation algorithm.

[0024] Preferably, step S3 further includes:

[0025] If the target is not detected for consecutive frames or the prediction deviation exceeds the threshold, pause the region of interest (ROI) analysis and trigger a global scan. Re-capture the target or discover new targets through the global scan, and re-initialize or terminate the tracking according to the scan results.

[0026] Preferably, the conditions for triggering a global scan include but are not limited to:

[0027] Triggering at a fixed time interval; Triggering by a target loss event; Triggering by a scene mutation event.

[0028] On the other hand, an auditing system based on the fast video auditing method based on temporal prediction as described above is provided, including:

[0029] An initialization module for performing global scanning, target detection, and initial state extraction;

[0030] A temporal prediction module for predicting the target state of the current frame based on historical states;

[0031] An ROI generation module for dynamically delimiting the region of interest;

[0032] A local analysis module for performing target analysis tasks within the ROI;

[0033] A state correction module for fusing prediction and measurement data to update the model;

[0034] A fault tolerance control module for handling target loss events and triggering a global scan.

[0035] Preferably, the temporal prediction module supports multi-model switching, including but not limited to: Kalman filtering, particle filtering, and LSTM networks, and dynamically selects the optimal model according to the scene complexity.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. Greatly improve the auditing efficiency: By concentrating the analysis and calculation on the predicted region of interest (ROI) instead of full-frame processing, the amount of calculation required for each frame is significantly reduced, thus accelerating the overall speed of video auditing;

[0038] 2. Significantly reduce computing resource consumption: Since only complex analyses such as object detection and recognition are performed on the ROI, redundant calculations for a large number of background or irrelevant regions in the video frames are avoided, effectively saving hardware resources such as processors and memory.

[0039] 3. Stronger adaptability and more obvious advantages for high-resolution videos: As the video resolution increases, the total number of pixels grows exponentially, resulting in a sharp increase in the computational burden of traditional full-frame analysis. By focusing on the ROI, the computational amount of this method is basically related to the size of the object and has little relationship with the growth of the overall resolution. Therefore, the higher the resolution, the more prominent the advantages of this method in terms of efficiency improvement and resource saving.

[0040] 4. Continuously optimize the prediction accuracy: After successfully finding the object within the ROI, the latest measurement state is used to update the time-series state prediction model, achieving continuous tracking of the object state and iterative optimization of the prediction model, which helps to more accurately predict the object position and define the ROI in subsequent frames. Brief Description of the Drawings

[0041] Figure 1 is the flowchart of the method of the present invention;

[0042] Figure 2 , where A is an example diagram of the initial detection frame of the present invention surrounding the detected face; B is the present invention's Figure 2 ROI example diagram generated using Alpha-Beta prediction based on A in; C is the present invention's Figure 2 Expanded ROI example diagram intercepted based on B in; D is the present invention's Figure 2 Example diagram of the result of face recognition within C in and the face position framed;

[0043] Figure 3 is a visualization example diagram of the complete process of prediction-identification-prediction based on the Alpha-Beta model of the present invention, where A, B, C, D, E, F are Figure 3 Specific process diagrams of;

[0044] Figure 4 , where A is an example diagram of initializing the particle filter of the present invention; B is the present invention's Figure 4 ROI of interest expanded based on A in; C is the present invention's Figure 4 Example diagram of intercepting the ROI of interest based on B in; D is the present invention's Figure 4 Example diagram of the result of face recognition within C in and the face position framed;

[0045] Figure 5 is a visualization example diagram of the complete process of prediction-identification-prediction based on the particle filter model of the present invention, where A, B, C, D, E, F areFigure 5 Specific process diagram;

[0046] Figure 6 is a visualization example diagram of the complete process of the target loss and recapture based on the particle filter model of the present invention, where A, B, C, and D are Figure 6 Specific process diagram;

[0047] Figure 7 is a schematic diagram of the system structure of the present invention. Specific implementation manners

[0048] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and not to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by this application.

[0049] In the present invention, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only relationship terms determined for the convenience of describing the structural relationship of each component or element of the present invention, and do not specifically refer to any component or element in the present invention, and should not be construed as a limitation to the present invention.

[0050] Embodiment 1:

[0051] As Figure 1 shown, this embodiment provides a fast video review method based on time series prediction using the Alpha-Beta filter as a time series state prediction model, including the following steps:

[0052] Step S1: Construct a time series state prediction model, perform a global scan on the initial frame of the video stream, and call the target detection algorithm to identify and locate the target;

[0053] Step S2: Extract the state information of the target in the initial frame to form an initial state vector , and initialize the time series state prediction model;

[0054] Step S3: Use the initialized time series state prediction model to perform iterative loop processing on each frame after the initial frame to complete video review.

[0055] Among them, steps S1-S2 can be regarded as the "initialization" stage (cold start process) in this embodiment. Step S1 can be regarded as the "target capture and initial state extraction" stage, and step S2 can be regarded as the "prediction model initialization" stage. In the "target capture and initial state extraction" stage and the "prediction model initialization" stage, the following are specifically executed:

[0056] First, detect the target of interest through a global image scanning method (such as a conventional object detection algorithm). Once the target is detected, as Figure 2 shown by the initial detection box in A, which surrounds the detected face. In the first frame of the video, its initial measurement state includes the center position and size , which are used to initialize the state vector of the Alpha-Beta filter . In this embodiment, the state vector contains the position, size, and their respective rates of change (velocities) of the target. Initially, the position and size components are set to the measured values, while all rate-of-change components (velocities , , and the rate of change of size , ) are usually set to zero.

[0057] Step S3 includes:

[0058] S31: Define each frame after the initial frame as the current frame , and based on the corrected state of the previous frame, predict the state of the current frame through the temporal state prediction model; ("Temporal state prediction" stage)

[0059] S32: According to the position information and size information of the predicted state , generate the region of interest in the current frame. The size of the region of interest is the product of the predicted size and the magnification factor , where ; ("Region of interest (ROI) generation" stage)

[0060] S33: Perform target analysis within the region of interest. The target analysis includes, but is not limited to: precise positioning, feature extraction, and classification recognition; ("Target analysis within ROI" stage)

[0061] S34: If the target is detected within the region of interest, obtain the measurement state of the target, and use to correct the temporal state prediction model and input the posterior state as the input for the next frame prediction. ("State update and iteration preparation" stage)

[0062] In the "Temporal state prediction" stage, specifically execute:

[0063] For each subsequent frame , use the corrected state , the Alpha-Beta filter predicts the target state of the current frame of the target . The prediction process is based on simplified dynamic models, such as constant velocity motion and constant size change models:

[0064] Predicted position: ;

[0065] Predicted size: ;

[0066] The velocity and size change rate are usually predicted to remain constant: ;

[0067] where is the time interval between frames.

[0068] In the "Region of Interest (ROI) Generation" stage, the following is specifically executed:

[0069] Based on the predicted target state in the predicted position and predicted size , combined with a preset or adaptive magnification factor (e.g., 1.8), a predicted Region of Interest (ROI) is generated. This ROI is expected to contain the actual position of the target in the current frame. (As shown in Figure 2 B, the purple dashed box is the ROI generated based on Alpha-Beta prediction, and the yellow dashed line is the ROI after expansion using the expansion factor)

[0070] In the "Target Analysis within ROI" stage, the following is specifically executed:

[0071] As shown in Figure 2 C, face recognition scans the face within this area, that is: the target detection algorithm is only executed within the predicted ROI generated in the "Region of Interest (ROI) Generation" stage, rather than scanning the entire image, significantly reducing the computational load;

[0072] If the target is successfully detected within the ROI (as shown in Figure 2 D, the green box is the target detected within the ROI), then the current measurement state is obtained.

[0073] In the "State Update and Iteration Preparation" stage, the following is specifically executed:

[0074] If the target is found within the ROI, then the measurement value and the predicted value are used to correct (update) the state of the Alpha-Beta filter. The update process is as follows:

[0075] Calculate the measurement residuals (the difference between the observed value and the predicted value):

[0076] (Calculate separately for each component);

[0077] Update the state:

[0078] ;

[0079] ;

[0080] (Similarly, use and to update the size and its change rate), where 、 、 、 are preset filter parameters, and the corrected state will be used for the prediction of the next frame .

[0081] Finally, for the remaining video frames , repeat the above "initialization" stage to "state update and iteration preparation" stage, and perform loop processing until the end of the video, as shown in Figure 3 A, B, C, D, E, F in, which are visualization examples of some of the frames.

[0082] Example 2:

[0083] This example provides a fast video review method based on temporal prediction using a particle filter as a temporal state prediction model, which specifically includes the following steps:

[0084] In the "initialization" stage, specifically execute:

[0085] Similar to Example 1, detect the initial target through global scanning (such as the initial detection box shown in Figure 4 A). After obtaining the initial measurement state , the system initializes particles (200 particles are initialized in this example). Each particle represents a state hypothesis of the initial state of the target. These particles are usually generated by sampling around according to a certain initial uncertainty distribution (such as a Gaussian distribution) (as shown in Figure 4 A in the appendix, where many particles are scattered around the initial target in the figure, and the purple box is the position of the prediction box). The initial weights of all particles are usually set to an equal value .

[0086] In the "Temporal State Prediction" stage, the following is specifically executed:

[0087] For subsequent frames , each particle (from the set of particles updated and resampled in the previous frame) is propagated (evolved) according to a preset dynamic model to obtain its predicted state in the current frame :

[0088] ;

[0089] The dynamic model is a simple random walk (i.e., the state at the previous moment plus a random perturbation).

[0090] In the "Region of Interest (ROI) Generation" stage, the following is specifically executed:

[0091] The predicted ROI is determined based on the collective positions and distributions of all predicted particles , for example, the weighted average state of these predicted particles can be calculated as the predicted center position and size of the target in the current frame , and then combined with the magnification factor , to generate the predicted ROI (as shown by the yellow dashed box in B of Figure 4 , which is the ROI predicted based on the particle cloud distribution, and the current state of the particle cloud may also be shown in the figure).

[0092] In the "Target Analysis within ROI" stage, the following is specifically executed:

[0093] The system only performs target detection within the predicted ROI (as shown by face recognition scanning for faces inside this region in C of Figure 4 ), and if a target is detected within this region, its measurement state is obtained;

[0094] If a target is successfully detected within the ROI (as shown by the green box in D of Figure 4 ).

[0095] In the "State Update and Iteration Preparation" stage, the following is specifically executed:

[0096] Weight calculation:

[0097] Based on the similarity (or "likelihood") between the measurement value and each predicted particle , the weight of the particle is updated , the closer the particle state is to the measurement value, the higher its weight. For example, the similarity can be calculated based on an exponential decay function of the difference between the two state vectors, and all weights are then normalized so that the sum is 1;

[0098] Resampling:

[0099] To prevent particle degeneracy (i.e., a few particles occupying almost all the weights), a number of particles are redrawn from the current particle set according to the updated weights to form a new particle set , particles with high weights have a greater probability of being selected multiple times. After resampling, the weights of the new particle set are usually reset to equal values , this step enables the particles to better predict the actual state of the target.

[0100] Finally, for the remaining video frames , repeat the above "initialization" phase to "state update and iteration preparation" phase, and perform loop processing until the video ends, as Figure 5 shown in A, B, C, D, E, F, which are visualization examples of some of the frames.

[0101] In addition, this embodiment also provides fault tolerance processing:

[0102] If the particle weights cannot be effectively updated within the predicted ROI (for example, all particles do not match any potential detections within the ROI), and this situation persists for M (preferably 7 in this example) frames, a global scan will be performed to attempt to re-capture the target, and the particle filter will be re-initialized based on the new strong detection results. As Figure 6 shown in the images of A, B, C, D, if no valid target can be detected continuously in the ROI, a global scan will be triggered to re-determine the target position and re-initialize the particle weights.

[0103] This preferred embodiment based on the particle filter can provide more robust ROI prediction for target tracking in complex scenarios through its probabilistic modeling of state uncertainty, thus supporting an efficient video review process.

[0104] As Figure 7 shown, this embodiment also provides an audit system based on the above fast video audit method based on temporal prediction, including:

[0105] An initialization module for performing global scanning, target detection, and initial state extraction;

[0106] A temporal prediction module for predicting the target state of the current frame based on historical states;

[0107] An ROI generation module for dynamically delimiting the region of interest;

[0108] A local analysis module for performing target analysis tasks within the ROI;

[0109] A state correction module for fusing prediction and measurement data to update the model;

[0110] The fault-tolerant control module is used to handle the target loss event and trigger a global scan.

[0111] The timing prediction module supports multi-model switching, including but not limited to: Kalman filtering, particle filtering, and LSTM networks, and dynamically selects the optimal model according to the scene complexity.

[0112] The above is a specific description of the preferred embodiment of the present invention, but the present invention is not limited to the described embodiment. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A fast video review method based on time series prediction, characterized in that It includes the following steps: Step S1: Construct a temporal state prediction model, perform a global scan on the initial frame of the video stream, and call a target detection algorithm to identify and locate the target; Step S2: Extract the state information of the target in the initial frame to form an initial state vector , and initialize the temporal state prediction model; Step S3: Use the initialized temporal state prediction model to perform iterative loop processing on each frame after the initial frame to complete video review; The said step S3 includes the following steps: S31: Define each frame after the initial frame as the current frame , based on the correction status of the previous frame , predict the status of the current frame through the timing status prediction model ; ; S32: According to the predicted state 's position information and size information, generate a region of interest in the current frame, and the size of the region of interest is the product of the predicted size and the magnification factor , where ; S33: Perform target analysis within the region of interest, and the target analysis includes but is not limited to: precise positioning, feature extraction, and classification recognition; S34: If the target is detected within the region of interest, obtain the measurement state of the target and use the corrected timing state prediction model to input the posterior state , which serves as the input for the next-frame prediction.

2. The fast video review method based on time series prediction according to claim 1, wherein, In the step S2, the state information of the target in the initial frame includes but is not limited to: the position information of the target of interest , the size information , and the motion parameters estimated through multi-frame data, where the motion parameters include but are not limited to the velocity information , the acceleration information .

3. The rapid video review method based on time series prediction according to claim 2, wherein The generation of the region of interest meets the following conditions: The width of the region of interest is: , and the height is: , where .

4. A fast video review method based on time series prediction according to claim 1, characterized in that The target analysis includes but is not limited to: Precisely locate the target within the region of interest using a deep learning-based detector; Extract the texture, color, or shape features of the target to enhance the tracking robustness; Identify the category attributes of the target through a classification model.

5. A fast video review method based on time series prediction according to claim 1, characterized in that, The model correction adopts the weighted fusion of the predicted state and the measured state by calculating the Kalman gain, or updates the weight parameters of the deep learning model through the backpropagation algorithm.

6. A fast video review method based on timing prediction according to claim 1, characterized in that The said step S3 further includes: If continuously frames do not detect a target or the prediction deviation exceeds the threshold, pause the region of interest analysis and trigger a global scan to recapture the target or discover a new target through the global scan, and re-initialize or terminate the tracking according to the scan results.

7. A fast video review method based on time series prediction according to claim 6, characterized in that The conditions for triggering a global scan include but are not limited to: Triggered at a fixed time interval; triggered by a target loss event; triggered by a scene mutation event.

8. An auditing system based on the fast video auditing method based on time series prediction as described in claim 1, characterized in that, It includes: An initialization module for performing global scan, target detection, and initial state extraction; A temporal prediction module for predicting the target state of the current frame based on historical states; An ROI generation module for dynamically delimiting the region of interest; A local analysis module for performing target analysis tasks within the ROI; A state correction module for fusing prediction and measurement data to update the model; A fault tolerance control module for handling target loss events and triggering a global scan.

9. An auditing system according to claim 8, characterized in that The said temporal prediction module supports multi-model switching, including but not limited to: Kalman filter, particle filter, and LSTM network, and dynamically selects the optimal model according to the scene complexity.

Images