A television tracking method combining twin networks and fast target optimization

Abstract

A television tracking method combining a twin network and a fast target optimization, a television tracking model is constructed for tracking shooting, the television tracking model includes a twin tracking network and a fast target optimization module, a preliminary tracking result is obtained from the tracking template and the search area by the twin tracking network, and then the tracking template used for tracking is adaptively updated by the fast target optimization module, online tracking and online updating of the tracker are realized. The application utilizes the efficient feature extraction capability of the twin network, can fully extract the features of the target and the search area, obtains the preliminary tracking result through similarity calculation, and simultaneously adopts the fast target optimization module, effectively solves the problem of low tracking robustness caused by complex target motion trajectory in television tracking.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision target tracking, and particularly relates to a television tracking method combining a twin network and fast target optimization. BACKGROUND

[0002] The television tracking technology is a technology that captures image information through a photoelectric camera, distinguishes a target from a background by using image processing means, calculates the position of the target in the image, and then controls a mechanical structure to drive the camera to always aim at the target according to the offset of the target coordinates from the center of the field of view or the movement law of the target. The technology makes the target always appear in the picture with a proper size through automatic rotation and zooming operation of the camera, and solves the problem that the tracking technology with a fixed visual angle in a wide outdoor environment cannot continuously track the target.

[0003] However, the target motion trajectory of the television tracking is not only the motion trajectory of the target itself in the real world, but also superimposes the motion trajectory of the camera itself, and the two are not simply relative motion, which makes the tracking trajectory of the target more complex. At the same time, occlusion, scale change, complex background and the like can cause the tracking method to make a mistake in judging the position of the target, control the camera to move in the wrong direction, thus intensifying the complexity of the target trajectory and causing the target to be lost. Therefore, the television tracking technology focuses on solving the tracking robustness caused by the complex target motion trajectory.

[0004] In the prior art, one class of methods is dedicated to researching the extraction of multi-layer features to generate a reliable feature expression, thereby improving the tracking accuracy, and another class of methods researches how to migrate general target tracking technology. However, these methods do not solve the problem of complex target motion trajectory of the television tracking. Therefore, targeted optimization for the complex motion trajectory of the television tracking and the problems of occlusion, scale change, complex background and the like helps to improve the robustness and stability of the tracking method. SUMMARY

[0005] The problem to be solved by the application is that, in view of the deficiencies in the existing tracking technology, a television tracking method combining a twin network and fast target optimization is proposed to improve the real-time performance and robustness of the tracker model.

[0006] The technical scheme of the application is a television tracking method combining a twin network and fast target optimization, a television tracking model is constructed for tracking shooting, the television tracking model includes a twin tracking network and a fast target optimization module, and the following steps are included.

[0007] Step S1: A data set is constructed for model training and testing.

[0008] Step S2: constructing a television tracking model, obtaining a preliminary tracking result from a tracking target template and a search area by a twin tracking network, and then performing result optimization by a fast target optimization module, the fast target optimization module including a motion estimation module, a global repositioning module and a template updating module, the motion estimation module performing motion estimation according to the preliminary tracking result through an optical flow algorithm and a position estimation algorithm, the optical flow algorithm calculating a deviation caused by camera displacement, and the position estimation algorithm estimating a position of the next frame through a historical tracking trajectory; the global repositioning module searching for a target in a whole image through a global search algorithm based on variance matching; and the template updating module evaluating tracking quality according to results of the motion estimation module and the repositioning module, and adaptively updating a tracking target template used for tracking;

[0009] Step S3: model training, training the model constructed in step S2, using an AdamW algorithm to train the model until loss converges, and obtaining trained model parameters;

[0010] Step S4: an online tracking stage:

[0011] Step S4.1: obtaining a label of a first frame of a video sequence as an initial tracking target template, selecting a region with a highest confidence score of a twin tracking network, and obtaining a preliminary tracking result;

[0012] Step S4.2: judging a fast target optimization operation to be performed according to a confidence score of the preliminary tracking result, a mechanical motion state of a camera and a historical tracking situation, optimizing the tracking result using the fast target optimization module, and updating a tracking template and a search area;

[0013] Step S4.3: repeating steps S4.1-S4.2, gradually calculating a tracking result of each frame, and thus realizing target tracking of the whole sequence.

[0014] The present application has the following advantages compared with the prior art:

[0015] (1) The present application fully utilizes the twin tracking network to fully mine the features of the tracking target and the search area, and uses the region recommendation network to improve the quality of the preliminary tracking result.

[0016] (2) The present application proposes a fast target optimization module, which effectively deals with the complex problem of target motion trajectory of television tracking by using a motion estimation module, a global search module and a template updating module, improves the accuracy and robustness of television tracking, and greatly improves the speed of television tracking, and can also achieve real-time level on edge devices with weak computing power. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1It is the overall flow chart of the present invention.

[0018] Figure 2 Schematic diagram of the region recommendation module of the present invention.

[0019] Figure 3 Schematic diagram of the motion estimation module of the present invention.

[0020] Figure 4 Schematic diagram of the global re-search module of the present invention.

[0021] Figure 5 Schematic diagram of the template update module of the present invention.

[0022] Figure 6 This is a comparison of the overall tracking performance of the present invention on the Anti-UAV dataset, where PR is the precision score and SR is the success rate score. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0024] like Figure 1 As shown in the overall tracking flow chart, the present invention provides a television tracking method combining twin networks and rapid target optimization, which includes the following steps.

[0025] Step S1: Construct a dataset and divide the corresponding data from the public Anti-UAV TV tracking dataset for model training and testing;

[0026] Step S2: Construct a TV tracking model, including a twin tracking network and a fast target optimization algorithm. The twin tracking network includes a feature extraction module and a region recommendation module. The fast target optimization algorithm includes a motion estimation module, a global matching module, and a template update module, as follows.

[0027] Step S2.1: The tracking target template and search area are used as inputs of the tracking model and are first processed by the Siamese tracking network;

[0028] Step S2.2: Construct a two-stream feature extraction module with shared weights and use the first three convolution layers of the lightweight ALEXNET to extract deep features of the target template and search area;

[0029] Step S2.3: Construct a region recommendation module, such as Figure 2As shown, using the region recommendation module in SiamFC++, the similarity of the features extracted in step S2.2 is calculated for the target and the template, and the candidate target region and its score are calculated through the quality head operation, the classification head operation and the regression head operation, and the region with the highest score is selected as the preliminary tracking result; the region recommendation network is used to calculate the correlation of the template and the target, and it is mapped to the same feature space to facilitate the head operation to obtain the preliminary tracking result, and the formula is as follows:

[0030] f i (z,x)=ψ i (φ(z))★ψ i (φ(x)),i∈{cls,reg} (1)

[0031] In the above formula, z and x represent the tracking target template picture and the search region picture respectively; φ() represents the backbone network used to extract the features of the two parts; ψ i () represents the feature mapping operation, cls represents the corresponding classification task, and reg represents the corresponding regression task; "★" represents the cross-correlation operation. After calculating the correlation information, the region recommendation will perform the classification and regression tasks, the input of the classification task is the cross-correlation result f cls (z,x) of the previous part for the classification task, and the corresponding image batch is divided into a positive batch or a negative batch. When classifying, if the feature map corresponding to the pixel is within the true value box, it is divided into a positive sample, otherwise it is divided into a negative sample. The regression task takes the cross-correlation result f reg (z,x) for the regression task as input to predict the boundary position, which can be represented by a four-tuple:

[0032]

[0033] In the above formula, s represents the total step of the backbone network; (x0,y0) and (x1,y1) represent the distance of point (x,y) from the left upper corner and the right lower corner of the corresponding true value boundary box respectively. The quality score is calculated from the four-tuple:

[0034]

[0035] The quality score is multiplied by the classification score to obtain the final confidence score, and the result with the highest confidence score is selected as the preliminary tracking result.

[0036] Step S2.4: Constructing a motion estimation module, as shown in Figure 3 The motion estimation module processes through an optical flow estimation algorithm and a position estimation algorithm. The optical flow estimation algorithm is used to calculate the offset caused by the displacement of the camera, and the position estimation algorithm estimates the position of the next frame through the historical tracking trajectory.

[0037] The position of the next frame is estimated through the historical tracking trajectory:

[0038] X k = AX k-1 + Q (4)

[0039] Z k = HX k + R (5)

[0040] P k = AP k-1 A T + Q (6)

[0041] K k = P k H T (HP k H T + R) -1 (7)

[0042] X' k = X k + K k (Z k - HX k ) (8)

[0043] P' k = P k - K k HP k (9)

[0044] The application discloses a position estimation method based on Kalman filtering, which estimates the target position of the next frame by using historical trajectory data and current observation results, and formula (4) and formula (5) are definitions of the target estimated position and the target motion position respectively, X k represents a prediction matrix, Z k represents an observation matrix, A represents a prediction state transition matrix, Q and R represent noise covariance matrices and system noise matrices, H represents an observation state transition matrix, formula (4) and formula (5) are initialized by using the tracking result of the first frame, and noise covariance matrices and system noise matrices are established, and a state transition matrix defined in advance is used for calculation; formula (6) describes a position estimation process, P k is an error matrix, which is calculated by a state transition matrix and a noise covariance matrix Q, formula (7), (8) and (9) describe a position estimation model updating process, K k represents a motion estimation gain, X' k and P' k represent an updated prediction matrix error matrix, and the finally updated prediction matrix is taken as a prediction result of the module.

[0045] Meanwhile, an offset generated by camera displacement is calculated by an optical flow algorithm:

[0046]

[0047] Equation (10) describes the calculation of the deviation of the motion prediction result caused by camera shake using the optical flow estimation method, V x and V y represent the picture offset in the horizontal and vertical directions, respectively, I x (q i ), I y (q i ) and I t (q i ) represent the partial derivatives of the pixel point q i with respect to the x-axis, the y-axis and the time axis, and finally the camera shake offset is superimposed on the position estimation result X ′ k to obtain the motion estimation result.

[0048] Step S2.5: Construct a global matching module to search for the lost target in the full image through a global search algorithm based on variance matching, as shown in Figure 4 In order to overcome the problem that the search area size of the twin tracking framework is limited and the target is easy to be lost, the application uses a global repositioning module based on variance matching. The input of the module is the previous tracking result sequence, the target template and the current frame. First, a stability evaluation algorithm is needed to determine whether global repositioning is needed. If the tracking result is stable, global repositioning will bring a lot of time loss and even disturb the original stable result. The stability evaluation algorithm comprehensively considers the time and space factors. If N consecutive frames are judged to be low quality in the time sequence, that is, the tracking position appears continuous jitter and drift phenomenon in the space, it is judged that the tracking is unstable or lost, and the search area position needs to be matched again. The jitter and drift can be determined by pre-designing parameters. In the matching process, the current complete image and the target template will be selected, and the sliding window principle will be used for sequential matching. The new target position of the target in the current frame is matched, and the search area position in the next frame in the twin tracking network is updated.

[0049] Step S2.6: Construct a template updating module to update the template used for tracking according to the tracking quality, as shown in Figure 5As shown, the tracking framework selects the target of the first frame of the sequence as the tracking template of the entire sequence, but the fixed template is unreliable in tracking, especially in television tracking, which involves camera zoom, object displacement, or target deformation. Using the initial template will obviously make the tracking gradually unstable. To solve this problem, the application uses a template updating module, which has three inputs: the current template used for tracking, the tracking quality sequence derived from the model, and the current mechanical state, which refers to the operating state of the camera, mainly including whether the camera has performed a zoom operation. In the template updating decision, the above three inputs are used to determine whether the template should remain unchanged, use the historical template or make a new template. If a new template needs to be made, a series of operations such as picture cropping, template feature extraction, window distance calculation, etc. are needed to get a new target tracking template, and then the newly made template is updated to the tracking template target for subsequent frame tracking.

[0050] Step S3: model training, load pre-trained model ALEXNET, repeat steps S2.1-S2.6, train the model using AdamW algorithm until the loss converges, and obtain the trained model parameters;

[0051] Step S4: online tracking phase, specifically as follows:

[0052] Step S4.1: obtain the label of the first frame of the video sequence as the initial tracking target, select the region with the highest score of the twin tracking network, and obtain the preliminary tracking result;

[0053] Step S4.2: determine the fast target optimization operation to be performed according to the confidence score of the twin tracking network, the mechanical motion state, and the historical tracking situation, optimize the tracking result using the fast target optimization method, and update the tracking template and the search area;

[0054] Step S4.3: repeat steps S4.1-S4.2 to gradually calculate the tracking result of each frame target, thereby realizing the target tracking of the entire sequence.

[0055] In order to verify the effectiveness of the tracking model of the application, comparative experiments were conducted on the widely used Anti-UAV dataset, and the experimental results were as follows: Figure 6Compared with the most advanced multi-modal tracking method at present, the accuracy and success rate indicators of the advanced SiamRPN++ of the application on Anti-UAV are significantly improved, the accuracy of the application on Anti-UAV is improved compared with the real-time multi-domain tracking network TransT, although the success rate indicator is slightly decreased, but the difference is not big, and the overall performance is still significantly improved. In addition, the running rate of the method of the application is about 110 FPS on the server and about 11 FPS on the edge device, which meets the real-time requirement.

Claims

1. A method of television tracking combining twin networks and fast target optimization, characterized in that: A TV tracking model is constructed for tracking shooting, and the TV tracking model comprises a twin tracking network and a fast target optimization module, and comprises the following steps: Step S1: constructing a data set for model training and testing; Step S2: constructing a TV tracking model, wherein the twin tracking network obtains a preliminary tracking result according to a tracking target template and a search area, and then the fast target optimization module is used for result optimization, the fast target optimization module comprises a motion estimation module, a global repositioning module and a template updating module, the motion estimation module performs motion estimation according to the preliminary tracking result through an optical flow estimation algorithm and a position estimation algorithm, the optical flow estimation algorithm calculates the offset caused by the camera displacement, and the position estimation algorithm estimates the position of the next frame through the historical tracking trajectory; the global repositioning module searches for the target in the whole image through a global search algorithm based on variance matching, and the template updating module evaluates the tracking quality according to the results of the motion estimation module and the repositioning module, and adaptively updates the tracking target template used for tracking; Step S3: model training, the model constructed in step S2 is trained, the model is trained using the AdamW algorithm until the loss converges, and the trained model parameters are obtained; Step S4: online tracking stage: Step S4.1: obtaining the label of the first frame of the video sequence as an initial tracking target template, selecting the area with the highest confidence score of the twin tracking network to obtain a preliminary tracking result; Step S4.2: judging the fast target optimization operation to be performed according to the confidence score of the preliminary tracking result, the mechanical motion state of the camera and the historical tracking situation, optimizing the tracking result using the fast target optimization module, and updating the tracking template and the search area; Step S4.3: repeating steps S4.1-S4.2 to gradually calculate the tracking result of each frame, thereby realizing the tracking of the target in the whole sequence.

2. The method of claim 1, wherein: The twin tracking network comprises a feature extraction module and a region recommendation module, the feature extraction module is a double-flow feature extraction structure with shared weights, the double-flow corresponds to a tracking target template and a search area respectively, and a lightweight ALEXNET network convolution is used as a backbone network to extract the deep features of the tracking target template and the search area; The region recommendation module calculates the similarity of the tracking target template and the candidate target in the search area according to the obtained deep features, and calculates the region of the candidate target obtained by searching and the confidence score thereof through quality head operation, classification head operation and regression head operation, selects the region with the highest score as the preliminary tracking result; in the model training of step S3, a pre-trained model ALEXNET is loaded first, and then the constructed TV tracking model is trained.

3. The method of claim 2, wherein: In the twin tracking network, the region recommendation network in the SiamFC++ method is used to calculate the similarity, and the similarity is mapped to the same feature space for the head operation of the quality head, the classification head and the regression head to obtain the preliminary tracking result, and the similarity calculation formula is as follows: (1) In the above formula, and respectively represent the tracking target template picture and the search area picture; represents the backbone network, represents the feature mapping operation, represents the corresponding classification task, represents the corresponding regression task; represents the cross-correlation operation, after the similarity information is calculated, the region recommendation model performs the classification and regression tasks, the input of the classification task is the cross-correlation result , and the corresponding image batch is divided into a positive batch or a negative batch, in the classification, if the feature map corresponding to the pixel is within the true value box, it is divided into a positive sample, otherwise it is divided into a negative sample; the regression task takes the cross-correlation result as input to predict the boundary position, which is represented by a four-tuple . (2) total step length of the backbone network; and representing a point distance to the left upper corner and the right lower corner of the corresponding ground truth bounding box, the quality score is calculated by a quadruple: (3) The final confidence score is obtained by multiplying the quality score and the classification score, and the result with the highest confidence is selected as the preliminary tracking result.

4. The method of claim 1, wherein: The motion estimation of the motion estimation module of the fast target optimization module is: The position of the next frame is estimated through the historical tracking trajectory: （4） （5） （6） （7） （8） （9） Based on Kalman filter, the historical trajectory data and the current observation result are used to estimate the target position of next frame, and formulas (4) and (5) are definitions of the target estimated position and the target motion position respectively, represents a prediction matrix, represents an observation matrix, represents a prediction state transition matrix, and represents a noise covariance matrix and a system noise matrix; represents an observation state transition matrix; The tracking result of the first frame is used to initialize the formula (4) and the formula (5), and a noise covariance matrix and a system noise matrix are established, and a pre-defined state transition matrix is used for calculation; the formula (6) describes a position estimation process, is an error matrix, which is obtained by multiplying the state transition matrix and the noise covariance matrix , and the formula (7) (8) (9) describe a position estimation model updating process, represents a motion estimation gain, and represent an updated prediction matrix and an error matrix, and the finally updated prediction matrix is used as a prediction result of the module. The offset caused by the camera displacement is calculated by the optical flow estimation algorithm: （10） Equation (10) describes the deviation of the motion prediction result caused by camera shake using the optical flow estimation method, and represent the horizontal and vertical directions of the picture offset, 、 and represent the partial derivatives of the pixel point with respect to the x-axis, y-axis and time axis, and finally the camera shake offset is superimposed on the position estimation result to obtain the motion estimation result.

5. The method of claim 1, wherein: The global repositioning module adopts a variance matching-based manner, and the input is a tracking result sequence, a tracking target template and a current frame. First, a stability evaluation algorithm is used to determine whether global repositioning is needed. The stability evaluation comprehensively considers time and space factors. If N consecutive frames are determined to be low quality in time sequence, or if the tracking position continuously shakes or drifts in space, it is determined that the tracking is unstable or lost, and the search area position needs to be re-matched. In the matching, the current complete image frame and the tracking target template are selected, and the sliding window principle is used for sequential matching. The new target position of the target is matched in the current frame, and the search area position of the next frame in the twin tracking network is updated.

6. The method of claim 1, wherein: The input of the template updating module includes three parts, the tracking target template used for tracking, the tracking quality sequence obtained by the twin tracking network, and the current mechanical state. The mechanical state refers to the operation state of the camera, including whether the camera is performing a zoom operation. The template updating decision determines whether the target tracking template should remain unchanged, use the historical template or make a new template according to the three input parts. If a new template needs to be made, the latest tracking result is processed, including picture cropping, template feature extraction, window distance calculation operations, to obtain a new target tracking template. Then, the newly made template updates the tracking template target, which is used for tracking of subsequent frames.

Images