Television tracking method combining twin network and fast target optimization

By combining the TV tracking model of twin network and fast target optimization module, problems such as complex target motion trajectory and occlusion are solved, the robustness and speed of tracking are improved, and efficient TV tracking effect is achieved.

CN119027693AActive Publication Date: 2024-11-26NANJING UNIV
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202310590319.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26
Estimated Expiration
2043-05-24

AI Technical Summary

Technical Problem

In TV tracking technology, the complex target motion trajectory, occlusion, scale changes and complex background lead to insufficient tracking robustness and stability, and the existing technology has failed to effectively solve these problems.

Method used

Combining twin networks and fast target optimization modules, a TV tracking model is built, including twin tracking networks, motion estimation modules, global relocation modules and template update modules. Through optical flow estimation algorithms, position estimation algorithms, global search algorithms and template update algorithms, Optimize tracking results to improve tracking accuracy and robustness.

Benefits of technology

It significantly improves the accuracy and robustness of TV tracking, improves tracking speed, and can also achieve real-time tracking on edge devices with weak computing power to meet the needs of real-time tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027693A_ABST
    Figure CN119027693A_ABST
Patent Text Reader

Abstract

A television tracking method combining a twin network and fast target optimization comprises the steps that a television tracking model is constructed for tracking shooting, the television tracking model comprises a twin tracking network and a fast target optimization module, and the twin tracking network obtains a preliminary tracking result according to a tracking template and a search area; and then, a tracking template used for tracking is adaptively updated by the rapid target optimization module, so that online tracking and online updating of the tracker are realized. According to the method, features of a target and a search area can be fully extracted by utilizing the efficient feature extraction capability of the twin network, a preliminary tracking result is obtained through similarity calculation, and meanwhile, the problem of relatively low tracking robustness caused by a complex target movement track in television tracking is effectively solved by adopting a rapid target optimization module.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision target tracking, and in particular relates to a television tracking method combining a twin network and fast target optimization. Background Art

[0002] Television tracking technology is a technology that captures image information through an optoelectronic camera, uses image processing methods to distinguish the target from the background, calculates the position of the target in the image, and then controls the mechanical structure to drive the camera to always aim at the target based on the offset between the target coordinates and the center of the field of view or the target's motion law. This technology uses automatic rotation and zoom operations of the camera to make the target always appear in the picture at an appropriate size, addressing the problem that fixed-viewing angle tracking technology is difficult to continuously track the target in a wide outdoor environment.

[0003] However, the target motion trajectory of TV tracking is not only the target's own motion trajectory in the real world, but also the camera's own motion trajectory. The two are not simply relative motions, which makes the target tracking trajectory more complicated. At the same time, occlusion, scale changes, complex backgrounds, etc. can cause the tracking method to make mistakes in the target position judgment, control the camera to move in the wrong direction, thereby increasing the complexity of the target trajectory and causing the target to be lost. Therefore, the focus of TV tracking technology is to solve the tracking robustness caused by the complexity of the target motion trajectory.

[0004] In the existing technology, one type of method is dedicated to studying the extraction of multi-layer features to generate a reliable feature expression to improve tracking accuracy, and another type of method studies how to migrate general target tracking technology. However, these methods do not specifically solve the problem of complex target motion trajectory in TV tracking. Therefore, targeted optimization of the complex motion trajectory of TV tracking, as well as occlusion, scale changes, and complex backgrounds can help improve the robustness and stability of the tracking method. Summary of the invention

[0005] The problem to be solved by the present invention is: in view of the shortcomings of the existing tracking technology, a TV tracking method combining a twin network and fast target optimization is proposed to improve the real-time and robustness of the tracker model.

[0006] The technical solution of the present invention is: a TV tracking method combining twin network and fast target optimization, constructing a TV tracking model for tracking shooting, the TV tracking model includes a twin tracking network and a fast target optimization module, including the following steps:

[0007] Step S1: construct a data set for model training and testing;

[0008] Step S2: construct a TV tracking model, and the twin tracking network obtains preliminary tracking results according to the tracking target template and the search area, and then the fast target optimization module optimizes the results. The fast target optimization module includes a motion estimation module, a global relocalization module and a template update module. The motion estimation module performs motion estimation according to the preliminary tracking results through an optical flow estimation algorithm and a position estimation algorithm. The optical flow algorithm calculates the offset caused by the camera displacement, and the position estimation algorithm estimates the position of the next frame through the historical tracking trajectory; the global relocalization module searches for the target in the entire image through a global search algorithm based on variance matching, and the template update module evaluates the tracking quality according to the results of the motion estimation module and the relocalization module, and adaptively updates the tracking target template used for tracking;

[0009] Step S3: Model training, training the model constructed in step S2, using the AdamW algorithm to train the model until the loss converges, and obtaining the trained model parameters;

[0010] Step S4: Online tracking phase:

[0011] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target template, select the area with the highest confidence score of the twin tracking network, and obtain the preliminary tracking result;

[0012] Step S4.2: Determine the fast target optimization operation to be performed based on the confidence score of the preliminary tracking result, the mechanical motion state of the camera, and the historical tracking situation, use the fast target optimization module to optimize the tracking result, and update the tracking template and search area;

[0013] Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire sequence.

[0014] The beneficial effects of the present invention are as follows: in view of the complexity of target trajectories in TV tracking technology, the present invention provides an efficient and accurate TV tracking solution, which has the following advantages over the prior art:

[0015] (1) The present invention makes full use of the twin tracking network to fully explore the characteristics of the tracking target and the search area, and uses the region recommendation network to improve the quality of the preliminary tracking results.

[0016] (2) The present invention proposes a fast target optimization module, which uses a motion estimation module, a global search module and a template update module to effectively deal with the complex problem of target motion trajectory in TV tracking, improves the accuracy and robustness of TV tracking, and greatly improves the speed of TV tracking, and can achieve real-time level even on edge devices with weak computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is the overall flow chart of the present invention.

[0018] Figure 2 It is a schematic diagram of the region recommendation module of the present invention.

[0019] Figure 3 It is a schematic diagram of a motion estimation module of the present invention.

[0020] Figure 4 Schematic diagram of the global re-search module of the present invention.

[0021] Figure 5 It is a schematic diagram of the template updating module of the present invention.

[0022] Figure 6 The overall tracking performance comparison of the present invention on the Anti-UAV dataset, where PR is the accuracy score and SR is the success rate score. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described in detail below in conjunction with the corresponding drawings:

[0024] like Figure 1 As shown in the overall tracking flow chart, the present invention provides a television tracking method combining twin networks and fast target optimization, which includes the following steps.

[0025] Step S1: construct a dataset and divide the corresponding data from the public Anti-UAV TV tracking dataset for model training and testing;

[0026] Step S2: construct a TV tracking model, including a twin tracking network and a fast target optimization algorithm. The twin tracking network includes a feature extraction module and a region recommendation module. The fast target optimization algorithm includes a motion estimation module, a global matching module and a template update module, as shown below.

[0027] Step S2.1: The tracking target template and the search area are used as the input of the tracking model and are first processed by the twin tracking network;

[0028] Step S2.2: Construct a two-stream feature extraction module with shared weights, and use the first three convolution layers of the lightweight ALEXNET to extract the deep features of the target template and the search area;

[0029] Step S2.3: Construct a region recommendation module, such as Figure 2As shown, the region recommendation module in SiamFC++ is used to calculate the similarity between the target and the template based on the features extracted in step S2.2, and the candidate target regions and their scores are calculated through quality head operation, classification head operation and regression head operation, and the region with the highest score is selected as the preliminary tracking result; the region recommendation network is used to calculate the correlation between the template and the target, and it is mapped to the same feature space to facilitate the head operation to obtain the preliminary tracking result. The formula is as follows:

[0030] f i (z,x)=ψ i (φ(z))★ψ i (φ(x)),i∈{cls,reg} (1)

[0031] In the above formula, z and x represent the tracking target template image and the search area image respectively; φ() represents the backbone network, which is used to extract the two parts of features; ψ i () represents feature mapping operation, cls represents the corresponding classification task, reg represents the corresponding regression task; "★" represents the cross-correlation operation. After calculating the correlation information, the region recommendation will perform classification and regression tasks. The input of the classification task is the cross-correlation result f for the classification task in the previous part. cls (z,x), and the corresponding image batch is divided into positive batch or negative batch. During classification, if the corresponding pixel of the feature map is within the true value box, it is classified as a positive sample, otherwise it is classified as a negative sample. The regression task will target the cross-correlation result f reg (z,x) is used as input to predict the boundary position, which can be represented by a four-tuple:

[0032]

[0033] In the above formula, s represents the total step length of the backbone network; (x0, y0) and (x1, y1) represent the distance from the point (x, y) to the upper left corner and lower right corner of the corresponding ground truth bounding box, respectively. The quality score is calculated from the four-tuple:

[0034]

[0035] The final confidence score is obtained by multiplying the quality score and the classification score, and the result with the highest confidence is selected as the preliminary tracking result.

[0036] Step S2.4: Construct a motion estimation module, such as Figure 3 As shown, the motion estimation module is processed by an optical flow estimation algorithm and a position estimation algorithm. The optical flow estimation algorithm is used to calculate the offset caused by the camera displacement, and the position estimation algorithm estimates the position of the next frame through the historical tracking trajectory.

[0037] The next frame position is estimated by historical tracking trajectory:

[0038] X k =AX k-1 +Q (4)

[0039] Z k =HX k +R (5)

[0040] P k =AP k-1 A T +Q (6)

[0041] K k =P k H T (HP k H T +R) -1 (7)

[0042] X′ k =X k +K k (Z k -HX k ) (8)

[0043] P′ k =P k -K k HP k (9)

[0044] The present invention discloses a position estimation method based on Kalman filtering, which uses historical trajectory data and current observation results to estimate the target position of the next frame. Formulas (4) and (5) are the definitions of the target estimated position and the target motion position, respectively. k Represents the prediction matrix, Z k represents the observation matrix, A represents the predicted state transfer matrix, Q and R represent the noise covariance matrix and the system noise matrix; H represents the observed state transfer matrix, and the tracking result of the first frame is used to initialize formula (4) and formula (5) and establish the noise covariance matrix and the system noise matrix, and the pre-defined state transfer matrix is ​​used for calculation; Formula (6) describes the position estimation process, P k is the error matrix, which is calculated by the state transfer matrix and the noise covariance matrix Q. Formulas (7), (8), and (9) describe the update process of the position estimation model. K k represents the motion estimation gain, X′ k and P′ k Represents the updated prediction matrix error matrix, and the final updated prediction matrix is ​​used as the prediction result of this module.

[0045] At the same time, the optical flow algorithm calculates the offset caused by the camera displacement:

[0046]

[0047] Formula (10) describes the deviation of motion prediction results caused by lens shake using the optical flow estimation method, V x and V y Represents the horizontal and vertical image offsets, I x (q i ), I y (q i ) and I t (q i ) represents pixel q i Partial differentiation of the x-axis, y-axis and time axis, and finally offset the camera shake Overlay on the position estimate X ′ k The result of motion estimation is obtained above.

[0048] Step S2.5: Construct a global matching module and search for the missing target in the whole image through a global search algorithm based on variance matching, such as Figure 4 As shown, in order to overcome the problem that the twin tracking framework has a limited search area size and is prone to losing the target, the present invention uses a global relocation module based on variance matching. The input of this module is the previous tracking result sequence, the target template and the current frame. First, it needs to go through a stability evaluation algorithm to determine whether global relocation is needed. If the tracking result is stable, global relocation will bring a lot of time loss and even disrupt the original stable result. The stability evaluation algorithm used integrates time and space factors. In terms of timing, if N consecutive frames are judged to be of low quality, that is, there is a continuous jitter and drift phenomenon in the tracking position in space, then the tracking is judged to be unstable or lost, and the search area position needs to be re-matched. The jitter and drift can be determined by pre-parameter design. During matching, the current complete image and target template will be selected, and the principle of sliding window will be used to match them in sequence. The new target position of the target is matched in the current frame, and the search area position of the next frame in the twin tracking network is updated.

[0049] Step S2.6: Construct a template update module to adaptively update the template used for tracking according to the tracking quality, such as Figure 5As shown, since the tracking framework selects the target of the first frame at the beginning of the sequence as the tracking template of the entire sequence, the fixed template is unreliable in tracking, especially in TV tracking, which involves camera zooming, object displacement, or target deformation. Using the initial template will obviously make the tracking gradually unstable. In order to solve this problem, the present invention uses a template update module, the input of which includes three parts: the template currently used for tracking, the tracking quality sequence obtained by the model, and the current mechanical state. The mechanical state refers to the operating state of the camera, mainly including whether the camera has performed a zoom operation. In the template update decision, it will be determined based on the above three inputs whether the template should remain unchanged, use the historical template or make a new template. The historical template refers to the template used by the tracking target. If a new template needs to be made, a series of operations such as picture cropping, template feature extraction, and window distance calculation are required for the current tracking result to obtain a new target tracking template, and then the newly made template is used to update the tracking template target for tracking subsequent frames.

[0050] Step S3: Model training, load the pre-trained model ALEXNET, repeat steps S2.1-S2.6, use the AdamW algorithm to train the model until the loss converges, and obtain the trained model parameters;

[0051] Step S4: Online tracking phase, specifically as follows:

[0052] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, select the area with the highest score of the twin tracking network, and obtain the preliminary tracking result;

[0053] Step S4.2: Determine the fast target optimization operation to be performed based on the confidence score of the twin tracking network, the mechanical motion state, and the historical tracking situation, use the fast target optimization method to optimize the tracking result, and update the tracking template and search area;

[0054] Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire sequence.

[0055] In order to verify the effectiveness of the tracking model of the present invention, a comparative experiment is conducted on the widely used Anti-UAV dataset. The experimental results are as follows: Figure 6As shown. Compared with the most advanced multimodal tracking method, the accuracy and success rate of the present invention on Anti-UAV are significantly improved compared with the advanced SiamRPN++. Compared with the real-time multi-domain tracking network TransT, the accuracy of the present invention on Anti-UAV is improved. Although the success rate index is slightly reduced, the difference is not large, and the overall performance is still significantly improved. In addition, the running speed of the present invention method is about 110FPS on the server and about 11FPS on the edge device, which meets the real-time requirements.

Claims

1. A TV tracking method combining twin networks and fast target optimization, characterized by Build a TV tracking model for tracking shooting. The TV tracking model includes a twin tracking network and a fast target optimization module, including the following steps: Step S1: construct a data set for model training and testing; Step S2: construct a TV tracking model, and the twin tracking network obtains preliminary tracking results according to the tracking target template and the search area, and then the fast target optimization module optimizes the results. The fast target optimization module includes a motion estimation module, a global relocalization module and a template update module. The motion estimation module performs motion estimation according to the preliminary tracking results through an optical flow estimation algorithm and a position estimation algorithm. The optical flow algorithm calculates the offset caused by the camera displacement, and the position estimation algorithm estimates the position of the next frame through the historical tracking trajectory; the global relocalization module searches for the target in the entire image through a global search algorithm based on variance matching, and the template update module evaluates the tracking quality according to the results of the motion estimation module and the relocalization module, and adaptively updates the tracking target template used for tracking; Step S3: Model training, training the model constructed in step S2, using the AdamW algorithm to train the model until the loss converges, and obtaining the trained model parameters; Step S4: Online tracking phase: Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target template, select the area with the highest confidence score of the twin tracking network, and obtain the preliminary tracking result; Step S4.2: Determine the fast target optimization operation to be performed based on the confidence score of the preliminary tracking result, the mechanical motion state of the camera, and the historical tracking situation, use the fast target optimization module to optimize the tracking result, and update the tracking template and search area; Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire sequence.

2. The TV tracking method combining twin network and fast target optimization according to claim 1, characterized in that The twin tracking network includes a feature extraction module and a region recommendation module. The feature extraction module is a dual-stream feature extraction structure with shared weights. The dual streams correspond to the tracking target template and the search area respectively. The lightweight ALEXNET network convolution is used as the backbone network to extract the deep features of the tracking target template and the search area. The region recommendation module calculates the similarity between the tracking target template and the candidate target in the search area based on the obtained deep features, and calculates the region where the searched candidate target is located and its confidence score through quality head operation, classification head operation and regression head operation, and selects the region with the highest score as the preliminary tracking result; in step S3 model training, first load the pre-trained model ALEXNET, and then train the constructed TV tracking model.

3. The TV tracking method combining twin network and fast target optimization according to claim 2, characterized in that In the twin tracking network, the region recommendation network in the SiamFC++ method is used to calculate the similarity and map it to the same feature space for the head operations of the quality head, classification head, and regression head to obtain preliminary tracking results. The similarity calculation formula is as follows: f i (z,x)=ψ i (φ(z))★ψ i (φ(x)),i∈{cls,reg} (1) In the above formula, z and x represent the tracking target template image and the search area image respectively; φ() represents the backbone network, ψ i () represents the feature mapping operation, cls represents the corresponding classification task, and reg represents the corresponding regression task; ″★″ represents the cross-correlation operation. After calculating the similarity information, the region recommendation model performs classification and regression tasks. The input of the classification task is the cross-correlation result f cls (z, x), and divide the corresponding image batch into positive batch or negative batch. During classification, if the corresponding pixel of the feature map is within the true value box, it is classified as a positive sample, otherwise it is classified as a negative sample. The regression task will cross-correlate the result f reg (z, x) as input and predict the boundary position, which is represented by a four-tuple (l * , t * ,r * , b * )express: s represents the total step length of the backbone network; (x0, y0) and (x1, y1) represent the distance of the point (x, y) from the upper left corner and the lower right corner of the corresponding ground truth bounding box, and the quality score is calculated by the four-tuple: The final confidence score is obtained by multiplying the quality score and the classification score, and the result with the highest confidence is selected as the preliminary tracking result.

4. The TV tracking method combining twin network and fast target optimization according to claim 1, characterized in that The motion estimation module of the fast target optimization module is: Estimate the next frame position by tracking the historical trajectory: X k =AX k-1 +Q (4) Z k =HX k +R (5) P k =AP k-1 From T +Q (6) K k =P k H T (HP k H T +R) -1 (7) X′ k =X k +K k (Z k -HX k ) (8) P′ k =P k -K k HP k (9) Based on Kalman filtering, the target position of the next frame is estimated using historical trajectory data and current observation results. Formulas (4) and (5) are the definitions of the target estimated position and the target motion position, respectively. k Represents the prediction matrix, Z k represents the observation matrix, A represents the predicted state transfer matrix, Q and R represent the noise covariance matrix and the system noise matrix; H represents the observation state transfer matrix; Use the tracking result of the first frame to initialize formula (4) and formula (5), and establish the noise covariance matrix and system noise matrix, and use the pre-defined state transfer matrix for calculation; Formula (6) describes the position estimation process, P k is the error matrix, which is calculated by the state transfer matrix and the noise covariance matrix Q. Formulas (7), (8), and (9) describe the update process of the position estimation model. K k represents the motion estimation gain, X′ k and P′ k Represents the updated prediction matrix and error matrix, and the final updated prediction matrix is ​​used as the prediction result of this module; The offset caused by the camera displacement calculated by the optical flow algorithm: Formula (10) describes the deviation of motion prediction results caused by lens shake using the optical flow estimation method, V x and V y Represents the horizontal and vertical image offsets, I x (q i ), I y (q i ) and I t (q i ) represents pixel q i Partial differentiation of the x-axis, y-axis and time axis, and finally offset the camera shake Superimposed on the position estimation result X′ k The result of motion estimation is obtained above.

5. The TV tracking method combining twin network and fast target optimization according to claim 1 is characterized in that the global The relocalization module adopts a variance matching-based approach. Its input is the tracking result sequence, the tracking target template and the current frame. It first passes through a stability evaluation algorithm to determine whether global relocalization is needed. The stability evaluation adopted integrates time and space factors. If N consecutive frames are judged to be of low quality in time sequence, that is, there is continuous jitter and drift in the tracking position in space, then the tracking is judged to be unstable or lost, and the search area position needs to be re-matched. During matching, the current complete image frame and the tracking target template are selected, and the sliding window principle is used to match them in sequence. The new target position is matched in the current frame, and the search area position of the next frame in the twin tracking network is updated.

6. The TV tracking method combining twin network and fast target optimization according to claim 1, characterized in that The input of the template update module includes three parts: the tracking target template currently used for tracking, the tracking quality sequence obtained by the twin tracking network, and the current mechanical state. The mechanical state refers to the operating state of the camera, including whether the camera is zooming. The template update decision is based on the three parts of the input to determine whether the target tracking template should remain unchanged, use the historical template, or create a new template. If a new template is required, the latest tracking result is processed, including screen cropping, template feature extraction, and window distance calculation operations to obtain a new target tracking template. The newly created template is then used to update the tracking template target for tracking subsequent frames.

Citation Information

Patent Citations

  • Moving target tracking method applied to complex background

    CN106296732A

  • Twin network tracking method based on self-adaptive template updating

    CN112132856A

  • Infrared weak and small target tracking method based on optical flow and Kalman filtering

    CN114066939A

  • Twin network target tracking method combining template updating and trajectory prediction

    CN115953570A

  • Automatic target point tracing method for electro-optical sighting system

    US20190162507A1