A video annotation method and system based on Kalman filtering

By applying Kalman filtering and selective search algorithms in video annotation, using the motion relationship information between video frames, the problem of low video annotation efficiency in the prior art is solved, and efficient video annotation is achieved, with an efficiency improvement of 30%- 80%.

CN113538528BActive Publication Date: 2024-11-29AISINO CORPORATION +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202110624567.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-04
Publication Date
2024-11-29
Estimated Expiration
2041-06-04

AI Technical Summary

Technical Problem

Existing video annotation software is inefficient when processing a large number of duplicate objects or video annotations, especially traditional methods cannot effectively utilize the motion relationship information between video frames, resulting in insufficient labeling efficiency and speed.

Method used

Using a video annotation method based on Kalman filter, combined with a Kalman filter and a selective search algorithm, the motion relationship information between video frames is used to annotate video frames, and parameters are performed through the state transition matrix and noise matrix of the Kalman filter. Update, the selective search algorithm finds candidate boxes and performs submission and comparison matches to determine the target box.

Benefits of technology

It significantly improves the video labeling efficiency, can improve the efficiency of 30%-80%, and significantly speeds up the labeling speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113538528B_ABST
    Figure CN113538528B_ABST
Patent Text Reader

Abstract

The present invention discloses a video annotation method and system based on Kalman filtering, including: initializing and setting the parameters of the Kalman filter; manually determining the starting frame of the position of the object to be annotated, and updating the Kalman filter using the coordinates of the starting frame; using the Kalman filter to predict the prediction frame of the position of the object to be annotated in the next frame; using a selective search algorithm to find all candidate frames containing the object to be annotated in the picture of the current frame; matching the prediction frame of the position of the object to be annotated with all candidate frames containing the object to be annotated, determining the best matching candidate frame, using the best matching candidate frame to update the prediction frame of the object to be annotated in the current frame, and annotating according to the updated prediction frame. Compared with the traditional annotation method, the present invention greatly improves the efficiency of video annotation, can improve the efficiency by 30%-80%, and effectively speeds up the annotation speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video annotation, and more specifically, to a video annotation method and system based on Kalman filtering. Background Art

[0002] In deep learning, data plays a key role in the prediction results of neural network models. Data is the "knowledge" learned by neural networks. Usually a large amount of data is needed to train the neural network to see obvious results. Data labeling is a time-consuming and labor-intensive task, so improving labeling efficiency is a key point.

[0003] At present, the commonly used data annotation software in the field of target detection include labelimg, cvat, labeling wizard, etc. Among them, labelimg and labeling wizard only have basic data annotation functions, and cvat is a more comprehensive annotation system that supports video annotation, but it still has many shortcomings, such as not supporting the prediction of common objects, slow annotation of static objects in videos, etc., especially when there are a large number of repeated objects in a picture or when annotating videos, the efficiency is extremely low.

[0004] Therefore, a method that can quickly and accurately perform video annotation is needed. Summary of the invention

[0005] The present invention proposes a video annotation method and system based on Kalman filtering to solve the problem of how to quickly and accurately perform video annotation.

[0006] In order to solve the above problem, according to one aspect of the present invention, a video annotation method based on Kalman filtering is provided, the method comprising:

[0007] Step 1, initializing the parameters of the Kalman filter;

[0008] Step 2, updating the parameters of the Kalman filter in sequence according to the starting boxes of the positions of the objects to be marked that are manually determined on the pictures of the preset number of frames;

[0009] Step 3, using the updated Kalman filter to predict the prediction box of the location of the object to be labeled in the current frame;

[0010] Step 4, using a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame;

[0011] Step 5: Match the predicted box of the object to be labeled in the current frame with all candidate boxes, determine the best matching candidate box, and use the best matching candidate box as the target box of the object to be labeled in the current frame.

[0012] Preferably, the parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

[0013] Preferably, the updating of the parameters of the Kalman filter in sequence according to the starting frame of the position of the object to be labeled which is manually determined on the picture of the preset number of frames includes:

[0014] S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including:

[0015] S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively;

[0016] S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ;

[0017] S213, P1 - , R, H, H T Substituting into formula (3) we get K1;

[0018] S214, will Substituting K1, z1, and H into equation (4) yields

[0019] S215, will Substituting K1, H, and I into formula (5) yields P1;

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] Among them, A is the state transfer matrix of the Kalman filter, and the value of A is:

[0026]

[0027] Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is:

[0028]

[0029] Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a fixed value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is:

[0030]

[0031] S22, using the optimal estimate of the state variables and the noise covariance corresponding to the current frame image in turn to calculate the optimal estimate of the state variables and the noise covariance corresponding to the next frame image, until the iterative update of the Kalman filter parameters at time k=n is completed, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is the preset number of frames.

[0032] Preferably, the step of using the updated Kalman filter to predict the prediction frame of the location of the object to be labeled in the current frame includes:

[0033] Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1 They are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box at the position of the object to be labeled on the n+1th frame image at time k=n+1.

[0034] Preferably, the step of searching for all candidate boxes containing the object to be labeled in the picture of the current frame by using a selective search algorithm comprises:

[0035] Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame. n A traversal comparison is performed to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of the candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, thereby determining all candidate boxes containing the object to be labeled in the current frame.

[0036] Preferably, the preset conditions include:

[0037] IOU1>K1 (7)

[0038]

[0039] Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-combination ratio, S 1交For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

[0040] Preferably, matching the predicted frame of the object to be labeled at the location of the current frame image with all candidate frames to determine the best matching candidate frame includes:

[0041] Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image;

[0042] When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box;

[0043] When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

[0044] Preferably, the method further comprises:

[0045] The Kalman filter is updated using the target box of the object to be labeled in the current frame image, and the calculation is returned to step 3 to recalculate until the prediction result does not meet the preset data labeling requirements, and the starting box of the object to be labeled is manually determined again, and the Kalman filter is updated according to the preset third frame number threshold using the manually determined starting box of the object to be labeled, and the calculation is returned to step 3 to recalculate until all frames are labeled.

[0046] According to another aspect of the present invention, a video annotation system based on Kalman filtering is provided, the system comprising:

[0047] An initialization unit, used for initializing the parameters of the Kalman filter;

[0048] A first updating unit, configured to update the parameters of the Kalman filter in sequence according to a start frame of a position of an object to be labeled that is manually determined on a picture of a preset number of frames;

[0049] A prediction unit, used to predict a prediction box of the position of the object to be labeled in the current frame by using the updated Kalman filter;

[0050] A candidate box determination unit, used to use a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame;

[0051] The labeling unit is used to match the predicted frame of the object to be labeled in the current frame image with all candidate frames, determine the best matching candidate frame, and use the best matching candidate frame as the target frame of the object to be labeled in the current frame image.

[0052] Preferably, the parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

[0053] Preferably, the first updating unit sequentially updates the parameters of the Kalman filter according to the starting frame of the position of the object to be labeled manually determined on the picture of the preset number of frames, including:

[0054] S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including:

[0055] S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively;

[0056] S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ;

[0057] S213, P1 - , R, H, H T Substituting into formula (3) we get K1;

[0058] S214, will Substituting K1, z1, and H into equation (4) yields

[0059] S215, will Substituting K1, H, and I into formula (5) yields P1;

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] Among them, A is the state transfer matrix of the Kalman filter, and the value of A is:

[0066]

[0067] Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is:

[0068]

[0069] Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a fixed value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,szk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is:

[0070]

[0071] S22, using the optimal estimate of the state variables and the noise covariance corresponding to the current frame image in turn to calculate the optimal estimate of the state variables and the noise covariance corresponding to the next frame image, until the iterative update of the Kalman filter parameters at time k=n is completed, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is the preset number of frames.

[0072] Preferably, the prediction unit predicts the prediction frame of the position of the object to be labeled in the current frame using the updated Kalman filter, including:

[0073] Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1 They are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box at the position of the object to be labeled on the n+1th frame image at time k=n+1.

[0074] Preferably, the candidate box determination unit uses a selective search algorithm to search for all candidate boxes containing the object to be labeled in the picture of the current frame, including:

[0075] Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame. n A traversal comparison is performed to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of the candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, thereby determining all candidate boxes containing the object to be labeled in the current frame.

[0076] Preferably, the preset conditions include:

[0077] IOU1>K1 (7)

[0078]

[0079] Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-union ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

[0080] Preferably, the labeling unit matches the predicted frame of the object to be labeled at the location of the current frame image with all candidate frames to determine the most matching candidate frame, including:

[0081] Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image;

[0082] When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box;

[0083] When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

[0084] Preferably, the system further comprises:

[0085] The second updating unit is used to update the Kalman filter by using the target box of the object to be labeled in the current frame picture, and enter the prediction unit to recalculate, until the prediction result does not meet the preset data labeling requirements, manually determine the starting box of the object to be labeled, and use the re-manually determined starting box of the object to be labeled to update the Kalman filter according to the preset third frame number threshold, and enter the prediction unit to recalculate until all frames are labeled.

[0086] The present invention provides a video annotation method and system based on Kalman filtering, which effectively utilizes the existing motion relationship information between video frames, and combines the Kalman filter and the selective search algorithm to annotate the video frames. Compared with the traditional annotation method, the video annotation efficiency is greatly improved, and the efficiency can be increased by 30%-80%, which effectively speeds up the annotation speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0088] Figure 1 is a flowchart of a video annotation method 100 based on Kalman filtering according to an embodiment of the present invention;

[0089] Figure 2 is a flow chart of a video annotation method based on Kalman filtering according to an embodiment of the present invention;

[0090] Figure 3 Schematic diagram of the structure of a video annotation system 300 based on Kalman filtering according to an embodiment of the present invention. DETAILED DESCRIPTION

[0091] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.

[0092] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0093] Figure 1 FIG. 1 is a flow chart of a video annotation method 100 based on Kalman filtering according to an embodiment of the present invention. Figure 1As shown, the video annotation method based on Kalman filtering provided by the embodiment of the present invention effectively utilizes the existing motion relationship information between video frames, and combines Kalman filtering and selective search algorithms to annotate video frames. Compared with the traditional annotation method, the video annotation efficiency is greatly improved, and the efficiency can be improved by 30%-80%, which effectively speeds up the annotation speed. The video annotation method 100 based on Kalman filtering provided by the embodiment of the present invention starts from step 101. In step 101, the parameters of the Kalman filter are initialized.

[0094] Preferably, the parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

[0095] In the present invention, the parameters for initializing the Kalman filter are specifically:

[0096] (1) Initialize the optimal estimate of the state variables of the Kalman filter at the initial time:

[0097]

[0098] Initialize the noise covariance of the Kalman filter at the initial time:

[0099]

[0100] (3) Initialize the state transfer matrix of the Kalman filter:

[0101]

[0102] (4) Initialize the noise matrix of the Kalman filter:

[0103]

[0104] In step 102, the parameters of the Kalman filter are updated in sequence according to the start boxes of the positions of the objects to be labeled that are manually determined on the pictures of the preset number of frames.

[0105] Preferably, the updating of the parameters of the Kalman filter in sequence according to the start frame of the position of the object to be labeled which is manually determined on the picture of the preset number of frames includes:

[0106] S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including:

[0107] S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively;

[0108] S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ;

[0109] S213, P1 - , R, H, H T Substituting into formula (3) we get K1;

[0110] S214, will Substituting K1, z1, and H into equation (4) yields

[0111] S215, will Substituting K1, H, and I into formula (5) yields P1;

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] Among them, A is the state transfer matrix of the Kalman filter, and the value of A is:

[0118]

[0119] Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is:

[0120]

[0121] Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a constant value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is:

[0122]

[0123] S22, using the optimal estimate of the state variables and the noise covariance corresponding to the current frame image in turn to calculate the optimal estimate of the state variables and the noise covariance corresponding to the next frame image, until the iterative update of the Kalman filter parameters at time k=n is completed, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is the preset number of frames.

[0124] In the present invention, the method of updating the parameters of the Kalman filter in sequence according to the starting frame of each frame of the picture is:

[0125] (1) Calculate the first frame of the image at time k=1 P1, the specific calculation method is:

[0126] 1.1) Manually select the starting box box1 of the object to be labeled on the first frame of the image. According to the information of box1, the z1 value can be obtained. z1 is the observed input value at the time k=1, z1=[u z1 ,v z2 ,s z3 ,r z4 ],u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively;

[0127] 1.2) Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ;

[0128] 1.3, P1 - , R, H, H T Substituting into formula (3) we get K1;

[0129] 1.4), Substituting K1, z1, and H into equation (4) yields

[0130] 1.5), Substituting K1, H, and I into formula (5) yields P1;

[0131]

[0132]

[0133]

[0134]

[0135]

[0136] In formula (1) and formula (2), A is the state transfer matrix of the Kalman filter, and the value of A is as follows:

[0137]

[0138] Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is as follows:

[0139]

[0140] In formula (6), u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, respectively, and assume that the aspect ratio r k Always remain the same;

[0141] In formula (4) to formula (6), K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, and the values ​​of H and I are as follows:

[0142]

[0143]

[0144] In summary, the calculation can be obtained P1, thereby completing the iterative update of the Kalman filter parameters at time k = 1, so as to be used for the second frame of the picture at time k = 2;

[0145] (2) The calculation method is the same as (1), based on P1 calculates the second frame of the picture P2;

[0146] according to P2 calculates the third frame of the picture P3;……;According to P n-1 Calculate the nth frame of the picture P n , thereby completing the operation on the first n consecutive frames of images, that is, completing the iterative update of the Kalman filter parameters at time k=n, so as to predict the position where the object to be labeled appears in the n+1th frame of image.

[0147] In step 103, the updated Kalman filter is used to predict a prediction box at the location of the object to be labeled in the current frame.

[0148] Preferably, the predicting of the prediction frame of the position of the object to be labeled in the current frame by using the updated Kalman filter includes:

[0149] Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1 They are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box at the position of the object to be labeled on the n+1th frame image at time k=n+1.

[0150] In the present invention, k=n+1 is taken, and the Substituting into formula (1.1), we can predict the predicted box of the object to be labeled on the n+1th frame image: u n+1 , v n+1 ,s n+1 , r n+1 They are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box of the object to be labeled on the n+1th frame image at the moment k=n+1.

[0151] In step 104, a selective search algorithm is used to search for all candidate boxes containing the object to be labeled in the picture of the current frame.

[0152] Preferably, the step of using a selective search algorithm to search for all candidate boxes containing the object to be labeled in the picture of the current frame includes:

[0153] Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame.n A traversal comparison is performed to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of the candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, thereby determining all candidate boxes containing the object to be labeled in the current frame.

[0154] Preferably, the preset conditions include:

[0155] IOU1>K1 (7)

[0156]

[0157] Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-combination ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

[0158] In the present invention, a threshold w is input to the Selective Search algorithm to obtain w candidate boxes, and the obtained w candidate boxes are compared with the box n Perform traversal comparison. If a candidate box that meets the preset conditions is found, the number of candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found; where box n Indicates that the starting box of the object to be labeled is manually selected on the nth frame image. n The information can be obtained from n Value, z n =[u zn ,v zn ,s zn ,r zn ],u zn ,v zn ,s zn ,r zn Box n The horizontal coordinate, vertical coordinate, area and aspect ratio of .

[0159] Optionally, the preset condition is: IOU1>K1, K1 is 0.7, and the calculation method of the intersection-over-union ratio IOU1 is:

[0160]

[0161] Among them, IOU1 is the candidate box and box n The intersection-over-combination ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union.

[0162] In step 105, the predicted box of the position of the object to be labeled in the current frame image is matched with all candidate boxes, the best matching candidate box is determined, and the best matching candidate box is used as the target box of the object to be labeled in the current frame image.

[0163] Preferably, matching the predicted frame of the object to be labeled at the location of the current frame image with all candidate frames to determine the best matching candidate frame includes:

[0164] Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image;

[0165] When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box;

[0166] When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

[0167] Preferably, the method further comprises:

[0168] The Kalman filter is updated using the target box of the object to be labeled in the current frame image, and the calculation is returned to step 3 to recalculate until the prediction result does not meet the preset data labeling requirements, and the starting box of the object to be labeled is manually determined again, and the Kalman filter is updated according to the preset third frame number threshold using the manually determined starting box of the object to be labeled, and the calculation is returned to step 3 to recalculate until all frames are labeled.

[0169] In the present invention, the object box A predicted by Kalman filtering is matched with the boxes of all candidate boxes found by the Selective Search algorithm to find the best matching candidate area as a new prediction box. In addition, during the matching process, the intersection-over-union (IOU) of the predicted box where the object to be labeled is located and each candidate box where the object to be labeled exists is calculated, including:

[0170]

[0171] Among them, IOU2 is the intersection-over-union ratio between the predicted box and all candidate boxes, S 2交 is the area of ​​the intersection of the prediction box and the candidate box, S 2并 is the area of ​​the union of the prediction box and the candidate box.

[0172] When the number of candidates with an IoU greater than a preset IoU threshold of 0.8 is less than or equal to a preset number threshold, the candidate box is selected as the best matching candidate box; when the number of candidates with an IoU greater than a preset IoU threshold is greater than the preset number threshold, the similarity between each candidate box with an IoU greater than the preset IoU threshold and the position prediction box of the object to be labeled is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the best matching candidate box. The preset number threshold can be set according to demand. For example, it can be set to 5.

[0173] If there are multiple target boxes that meet the best matching condition, the similarity between each candidate box and the predicted box is calculated, and the candidate box with the largest similarity is taken as the target box of the object to be labeled in the current frame image, i.e., the n+1 frame image, and the target box is used to update the parameters of the Kalman filter for use in calculating and updating the parameters of the Kalman filter for the image at time k=n+2.

[0174] In addition, the steps 103-105 are repeated for a number of consecutive frames after the n+2th frame (including the n+2th frame), until the prediction result does not meet the manually defined data labeling requirements, and the object to be labeled is manually selected on the m consecutive frames after the current frame to obtain a new starting frame, and the cycle is repeated, and the parameters of the Kalman filter are updated in sequence according to the new starting frame. The value of m is 2 or 3, which can ensure the accuracy of the subsequent prediction results without causing data redundancy.

[0175] The video annotation method based on Kalman filtering of the present invention effectively utilizes the existing motion relationship information between video frames, and combines the Kalman filter and the selective search algorithm to annotate the video frames. Compared with the traditional annotation method, the video annotation efficiency is greatly improved, and the efficiency can be improved by 30%-80%, which effectively speeds up the annotation speed.

[0176] like Figure 2 As shown, the video annotation method based on Kalman filtering of the present invention includes:

[0177] Step A: Initialize the parameters of the Kalman filter, including: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

[0178] Step B: Artificially select an object A as the object to be labeled, manually select object A on the first n consecutive frames of images, obtain the starting frame of object A, and update the parameters of the Kalman filter according to the starting frame of each frame of the image. and P k , as follows:

[0179] In the embodiment of the present invention, it is assumed that n=3=k, then:

[0180] Calculate the time k = 1 P1, thereby completing the iterative update of the Kalman filter parameters at time k = 1 for use in the picture at time k = 2. The specific calculation method is:

[0181] 1.1) Take k = 1, i.e. the first frame of the image, and manually select the starting frame box1 of object A on the first frame of the image. According to the information of box1, the z1 value can be obtained. z1 is the observed input value at the moment k = 1, z1 = [u z1 ,v z1 ,s z1 ,r z1 ],u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively.

[0182] 1.2) Substituting P0, A, and Q into the above equations (11) and (1), we obtain and P1 - ;

[0183] 1.3) P1 - , R, H, H T Substituting into formula (3) we get K1;

[0184] 1.4) Substituting K1, z1, and H into equation (4) yields

[0185] 1.5) Substituting K1, H, and I into equation (5) yields P1.

[0186] In summary, the calculation can be obtained and P1, thereby completing the iterative update of the Kalman filter parameters at time k = 1, and P1 are used to calculate and update the parameters of the Kalman filter for the image at time k = 2 Used when using with P2.

[0187] (2) Take k = 2, that is, the second frame of the picture. Manually select the starting box box2 of object A on the second frame of the picture. According to the information of box2, the z2 value can be obtained, z2 = [u z2 ,v z2 ,s z2 ,r z2 ],[u z2 ,v z2 ,s z2 ,r z2 ] are the horizontal coordinate, vertical coordinate, area and aspect ratio of box2 respectively. The calculation method is the same as (1), and the calculation can be obtained and P2, thereby completing the iterative update of the Kalman filter parameters at time k = 2, P2 and P3 are used to calculate and update the parameters of the Kalman filter for the image at time k = 3. Used when using with P3.

[0188] (3) Take k = 3, i.e. the third frame, and manually select the starting box box3 of object A on the third frame. According to the information of box3, the z3 value can be obtained, z3 = [u z3 ,v z3 ,s z3 ,r z3 ],u z3 ,v z3 ,s z3 ,r z3 are the horizontal coordinate, vertical coordinate, area and aspect ratio of box3 respectively. The calculation method is the same as (1), and we can get and P3, thereby completing the iterative update of the Kalman filter parameters at time k = 3, P3 and P4 are used to calculate and update the parameters of the Kalman filter for the image at time k = 4. Used when using .

[0189] In this way, the operation on the first three consecutive frames of images is completed, thereby completing the iterative update of the Kalman filter parameters at time k=3, so as to predict the position where the object A appears in the fourth frame of the image.

[0190] Step C: Use the Kalman filter with updated parameters to predict the position of object A in the n+1th frame image, and obtain the predicted frame of object A, specifically:

[0191] The (n+1)th frame picture in the embodiment of the present invention is the 4th frame picture.

[0192] Take k = 4, and convert the Substituting into formula (1), we can predict the predicted box of object A in the fourth frame: u4, v4, s4, and r4 are respectively the horizontal coordinate, vertical coordinate, area, and aspect ratio of the center position of the predicted box of object A at the moment k=4, i.e., the 4th frame.

[0193] In order to ensure the accuracy of the prediction box, step D and step E are performed.

[0194] Step D: Find all candidate boxes containing object A in the n+1th frame, specifically:

[0195] The (n+1)th frame picture in the embodiment of the present invention is the 4th frame picture.

[0196] The embodiment of the present invention uses the Selective Search algorithm for searching: input threshold value w=100 to the Selective Search algorithm, obtain 100 candidate boxes, and traverse and compare the obtained 100 candidate boxes with box3 in step B. If a candidate box that meets the preset condition can be found, the number of candidate boxes is the threshold value 100; otherwise, the threshold value 100 is incremented by 1 to obtain a new threshold value 101, and the new threshold value 101 continues to be traversed and compared in the fourth frame until a candidate box that meets the preset condition is found. The preset condition is IOU1>0.7.

[0197] Step E: Match the predicted box obtained in step C with all the candidate boxes in step D, and find the best matching candidate box as the target box of object A in the n+1th frame image. Specifically:

[0198] The n+1th frame of the embodiment of the present invention is the 4th frame, and the most matching judgment standard is: IOU2>0.8. If there are multiple candidate frames that meet the above-mentioned most matching conditions, the similarity between each candidate frame and the predicted frame is calculated, and the candidate frame with the largest similarity is taken as the target frame of object A on the 4th frame. The target frame is used to update the parameters of the Kalman filter for the picture at time k=5 when calculating and updating the parameters of the Kalman filter. The specific updating method is the same as step B, and the target frame is the prediction result.

[0199] The similarity calculation method adopts existing technology to perform calculation, for example, uses OpenCV template matching algorithm to perform calculation.

[0200] Step F: Repeat steps C, D and E for several consecutive frames after the n+2th frame (including the n+2th frame) until the prediction result does not meet the manually defined data labeling requirements. Manually select object A on the m consecutive frames after the current frame to obtain a new starting frame. Repeat this cycle and update the parameters of the Kalman filter in sequence according to the new starting frame. The specific updating method is the same as step B.

[0201] Human-defined data annotations, such as: the prediction box completely encloses the boundary of object A.

[0202] The experimental data show that the appropriate value of m is 2 or 3, which can ensure the accuracy of subsequent prediction results without causing data redundancy.

[0203] Step G: Repeat step F until all the pictures in the video are labeled.

[0204] Figure 3 FIG. 3 is a schematic diagram of the structure of a video annotation system 300 based on Kalman filtering according to an embodiment of the present invention. Figure 3 As shown, the video annotation system 300 based on Kalman filtering provided by the embodiment of the present invention includes: an initialization unit 301, a first updating unit 302, a prediction unit 303, a candidate box determination unit 304 and an annotation unit 305.

[0205] Preferably, the initialization unit 301 is used to initialize the parameters of the Kalman filter.

[0206] Preferably, the parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

[0207] Preferably, the first updating unit 302 is used to update the parameters of the Kalman filter in sequence according to the starting boxes of the positions of the objects to be labeled that are manually determined on the pictures of a preset number of frames.

[0208] Preferably, the first updating unit 302 updates the parameters of the Kalman filter in sequence according to the starting frame of the position of the object to be labeled manually determined on the picture of the preset number of frames, including:

[0209] S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including:

[0210] S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively;

[0211] S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ;

[0212] S213, P1 - , R, H, H T Substituting into formula (3) we get K1;

[0213] S214, will Substituting K1, z1, and H into equation (4) yields

[0214] S215, will Substituting K1, H, and I into formula (5) yields P1;

[0215]

[0216]

[0217]

[0218]

[0219]

[0220] Among them, A is the state transfer matrix of the Kalman filter, and the value of A is:

[0221]

[0222] Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is:

[0223]

[0224] Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a constant value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is:

[0225]

[0226] S22, using the optimal estimate of the state variables and the noise covariance corresponding to the current frame image in turn to calculate the optimal estimate of the state variables and the noise covariance corresponding to the next frame image, until the iterative update of the Kalman filter parameters at time k=n is completed, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is the preset number of frames.

[0227] Preferably, the prediction unit 303 is used to predict a prediction box at the location of the object to be labeled in the current frame by using an updated Kalman filter.

[0228] Preferably, the prediction unit 303 predicts the prediction frame of the position of the object to be labeled in the current frame using the updated Kalman filter, including:

[0229] Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1They are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box at the position of the object to be labeled on the n+1th frame image at time k=n+1.

[0230] Preferably, the candidate box determining unit 304 is used to use a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame.

[0231] Preferably, the candidate box determination unit 304 uses a selective search algorithm to search for all candidate boxes containing the object to be labeled in the picture of the current frame, including:

[0232] Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame. n A traversal comparison is performed to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of the candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, thereby determining all candidate boxes containing the object to be labeled in the current frame.

[0233] Preferably, the preset conditions include:

[0234] IOU1>K1 (7)

[0235]

[0236] Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-union ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

[0237] Preferably, the labeling unit 305 is used to match the position prediction box of the object to be labeled in the current frame image with all candidate boxes, determine the best matching candidate box, and use the best matching candidate box as the target box of the object to be labeled in the current frame image.

[0238] Preferably, the tagging unit 305 matches the predicted box of the object to be tagged at the location of the current frame image with all candidate boxes to determine the most matching candidate box, including:

[0239] Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image;

[0240] When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box;

[0241] When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

[0242] Preferably, the system further comprises:

[0243] The second updating unit is used to update the Kalman filter by using the target box of the object to be labeled in the current frame picture, and enter the prediction unit to recalculate, until the prediction result does not meet the preset data labeling requirements, manually determine the starting box of the object to be labeled, and use the re-manually determined starting box of the object to be labeled to update the Kalman filter according to the preset third frame number threshold, and enter the prediction unit to recalculate until all frames are labeled.

[0244] The video annotation system 300 based on Kalman filtering in the embodiment of the present invention corresponds to the video annotation method 100 based on Kalman filtering in another embodiment of the present invention, and will not be described in detail herein.

[0245] The invention has been described above with reference to a few embodiments. However, it is readily apparent to a person skilled in the art that other embodiments than the ones disclosed above are equally within the scope of the invention, as defined by the appended patent claims.

[0246] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / said / the [means, components, etc.]" are to be openly interpreted as at least one instance of the means, components, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not necessarily have to be performed in the exact order disclosed, unless explicitly stated otherwise.

[0247] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0248] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0249] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0250] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0251] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A video annotation method based on Kalman filtering, characterized in that: The method comprises: Step 1, initializing the parameters of the Kalman filter; Step 2, updating the parameters of the Kalman filter in sequence according to the starting boxes of the positions of the objects to be marked that are manually determined on the pictures of the preset number of frames; Step 3, using the updated Kalman filter to predict the prediction box of the location of the object to be labeled in the current frame; Step 4, using a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame; Step 5, matching the predicted frame of the object to be labeled in the current frame with all candidate frames, determining the best matching candidate frame, and using the best matching candidate frame as the target frame of the object to be labeled in the current frame; The updating of the parameters of the Kalman filter in sequence according to the start frame of the position of the object to be marked manually determined on the picture of the preset number of frames includes: S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including: S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively; S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and S213, will R, H, H T Substituting into formula (3) we get K1; S214, will Substituting K1, z1, and H into equation (4) yields S215, will Substituting K1, H, and I into formula (5) yields P1; Among them, A is the state transfer matrix of the Kalman filter, and the value of A is: Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is: Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a constant value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is: S22, sequentially using the optimal estimate of the state variable and the noise covariance corresponding to the current frame image to calculate the optimal estimate of the state variable and the noise covariance corresponding to the next frame image, until completing the iterative update of the Kalman filter parameters at time k=n, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is a preset number of frames; The method of using the updated Kalman filter to predict the prediction frame of the location of the object to be labeled in the current frame includes: Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1 are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box of the position of the object to be labeled on the n+1th frame image at time k=n+1; The method of using a selective search algorithm to search for all candidate boxes containing the object to be labeled in the picture of the current frame includes: Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame. n Perform traversal comparison to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, and all candidate boxes containing the object to be labeled in the current frame are determined; The preset conditions include: IOU1>K1 (7) Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-combination ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

2. The method according to claim 1, characterized in that The parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

3. The method according to claim 1, characterized in that The step of matching the predicted frame of the object to be labeled at the location of the current frame image with all candidate frames to determine the best matching candidate frame includes: Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image; When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box; When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

4. The method according to claim 1, characterized in that: The method further comprises: The Kalman filter is updated using the target box of the object to be labeled in the current frame image, and the calculation is returned to step 3 to recalculate until the prediction result does not meet the preset data labeling requirements, and the starting box of the object to be labeled is manually determined again, and the Kalman filter is updated according to the preset third frame number threshold using the manually determined starting box of the object to be labeled, and the calculation is returned to step 3 to recalculate until all frames are labeled.

5. A video annotation system based on Kalman filtering, characterized in that: The system comprises: An initialization unit, used for initializing the parameters of the Kalman filter; A first updating unit, configured to update the parameters of the Kalman filter in sequence according to a start frame of a position of an object to be labeled that is manually determined on a picture of a preset number of frames; A prediction unit, used to predict a prediction box of a position of the object to be labeled in the current frame by using the updated Kalman filter; A candidate box determination unit, used to use a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame; A labeling unit, used to match the predicted frame of the position of the object to be labeled in the current frame image with all candidate frames, determine the best matching candidate frame, and use the best matching candidate frame as the target frame of the object to be labeled in the current frame image; The first updating unit sequentially updates the parameters of the Kalman filter according to the starting frame of the position of the object to be labeled manually determined on the picture of the preset number of frames, including: S21, calculate the optimal estimate of the state variable of the first frame of the picture at time k=1 and noise covariance P1 for use in the second frame of the picture at time k=2, including: S211, based on the information of the starting box box1 of the position of the object to be labeled determined on the first frame image, the observed input value z1 = [u z1 ,v z1 ,s z1 ,r z1 ]; where u z1 ,v z1 ,s z1 ,r z1 They are the horizontal coordinate, vertical coordinate, area and aspect ratio of box1 respectively; S212, will Substituting P0, A, and Q into equations (1) and (2), we obtain and P1 - ; S213, will R, H, H T Substituting into formula (3) we get K1; S214, will Substituting K1, z1, and H into equation (4) yields S215, will Substituting K1, H, and I into formula (5) yields P1; Among them, A is the state transfer matrix of the Kalman filter, and the value of A is: Among them, Δt is the interval time or number of frames; A T is the transposed matrix of A; Q is the noise matrix of the Kalman filter; is the optimal estimate of the state variable of the Kalman filter at time k-1; P k-1 is the noise covariance of the Kalman filter at time k-1, indicating the mutual influence between the various state variables, P k-1 is a matrix with 7 rows and 7 columns; is the initial prediction value of the noise covariance of the Kalman filter at time k; is the preliminary estimate of the state variable of the Kalman filter at time k, and its value is: Among them, u k ,v k ,s k ,r k are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the prediction box at time k, are the change rates of the horizontal, vertical and area of ​​the prediction box at time k, and the aspect ratio r k is a constant value; K k is the Kalman gain coefficient at time k; is the optimal estimate of the state variables of the Kalman filter at time k; P k is the optimal estimate of the noise covariance of the Kalman filter at time k, which is used for the next iterative update; is the initial prediction value of the noise covariance of the Kalman filter at time k; z k is the observed input value at time k, z k =[u zk ,v zk ,s zk ,r zk ],u zk ,v zk ,s zk ,r zk are the horizontal coordinate, vertical coordinate, area and aspect ratio of the center position of the starting frame at time k; H T is the transposed matrix of H, H is the observation function; R is the measurement noise, R = 1000*I, I is the 4th-order unit matrix; the value of H is: S22, sequentially using the optimal estimate of the state variable and the noise covariance corresponding to the current frame image to calculate the optimal estimate of the state variable and the noise covariance corresponding to the next frame image, until completing the iterative update of the Kalman filter parameters at time k=n, so as to predict the position where the object to be labeled appears in the n+1th frame image; wherein n is a preset number of frames; The prediction unit predicts the prediction frame of the position of the object to be labeled in the current frame by using the updated Kalman filter, including: Take the current frame number as k=n+1, and replace the Substitute into formula (1) to predict the predicted box of the location of the object to be labeled on the n+1th frame image u n+1 , v n+1 ,s n+1 , r n+1 are respectively the horizontal coordinate, vertical coordinate, area and aspect ratio of the predicted box of the position of the object to be labeled on the n+1th frame image at time k=n+1; The candidate box determination unit uses a selective search algorithm to find all candidate boxes containing the object to be labeled in the picture of the current frame, including: Input the threshold w to the Selective Search algorithm, obtain w candidate boxes, and compare the obtained w candidate boxes with the starting box of the location of the object to be labeled in the previous frame. n Perform traversal comparison to determine all candidate boxes containing the object to be labeled in the current frame; if a candidate box that meets the preset conditions can be found, the number of candidate boxes is the threshold w; otherwise, the threshold w is incremented by 1 to obtain a new threshold w+1, and the new threshold w+1 continues to be traversed and compared in the n+1th frame until a candidate box that meets the preset conditions is found, and all candidate boxes containing the object to be labeled in the current frame are determined; The preset conditions include: IOU1>K1 (7) Among them, for any candidate box, IOU1 is the IOU between the candidate box and the box n The intersection-over-union ratio, S 1交 For the candidate box and box n The area of ​​the intersection, S 1并 For the candidate box and box n The area of ​​the union, K1 is the first intersection-over-union ratio threshold; wherein, if the candidate box satisfies IOU1>K1, it is determined that the candidate box meets the requirements.

6. The system according to claim 5, characterized in that The parameters of the Kalman filter include: the optimal estimate of the state variables of the Kalman filter at the initial time The noise covariance P0 of the Kalman filter at the initial moment, the state transfer matrix A of the Kalman filter, and the noise matrix Q of the Kalman filter.

7. The system according to claim 5, characterized in that The labeling unit matches the predicted frame of the object to be labeled at the location of the current frame image with all candidate frames to determine the most matching candidate frame, including: Calculate the intersection-over-union ratio of the predicted box and each candidate box at the location of the object to be labeled in the current frame image; When there is only one candidate box whose IoU is greater than a preset second IoU threshold, the candidate box is selected as the best matching candidate box; When there are at least two candidate boxes whose IoU ratio is greater than a preset second IoU ratio threshold, the similarity between each candidate box whose IoU ratio is greater than the preset IoU ratio threshold and the predicted box of the position of the object to be labeled in the current frame image is calculated respectively, and the candidate box corresponding to the maximum similarity is selected as the most matching candidate box.

8. The system according to claim 5, characterized in that The system further comprises: The second updating unit is used to update the Kalman filter by using the target box of the object to be labeled in the current frame picture, and enter the prediction unit to recalculate, until the prediction result does not meet the preset data labeling requirements, manually determine the starting box of the object to be labeled, and use the re-manually determined starting box of the object to be labeled to update the Kalman filter according to the preset third frame number threshold, and enter the prediction unit to recalculate until all frames are labeled.

Citation Information

Patent Citations

  • Multi-target behavior identification method and system for monitoring video

    CN110378259A

  • Human body behavior recognition method and system based on multi-target tracking

    CN110399808A