A dynamic fusion target tracking method
By using a dynamic fusion target tracking method, a single-stream target tracking model and attention mechanism are employed to reduce background information interference, thereby improving the accuracy and robustness of target tracking and reducing computational complexity.
Patent Information
- Application Number
- CN202311246691.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-26
- Publication Date
- 2025-11-07
- Estimated Expiration
- 2043-09-26
AI Technical Summary
Existing target tracking methods have shortcomings in terms of tracking accuracy, robustness and efficiency. In particular, single-stream and dual-stream trackers based on attention mechanisms have high computational costs and are severely affected by background information interference when processing the search area.
A dynamic fusion target tracking method is adopted, which utilizes a single-stream target tracking model based on an attention mechanism. The feature extraction network obtains the relationship information between the feature map of the search region and the fusion token, gradually reduces the number of tokens in the search region, and reduces background information interference through token fusion. The predicted target position is used for staged prediction.
It improves the accuracy and robustness of target tracking, reduces computational complexity and cost, and minimizes the contamination of target features by background information.
Smart Images
Figure CN117237672B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application relates to a dynamic fusion target tracking method and belongs to the technical field of computer vision. BACKGROUND
[0002] Target tracking is one of important application fields of computer vision. A target to be tracked is marked in the first frame of a video, and a target tracking algorithm can predict the position and size of the target in subsequent video frames. Based on this function, target tracking has a large number of applications in unmanned aerial vehicles, automatic driving, medical image processing and segmentation and the like. However, considering the complexity and diversity of tracking scenes, there is still much room for improvement in tracking accuracy, robustness and efficiency of existing trackers.
[0003] Most early target tracking methods are based on correlation filtering. The tracker based on correlation filtering has a high running speed, but the tracking accuracy is not ideal. With the development of deep learning in the field of computer vision, tracking methods based on deep learning begin to appear. Compared with the traditional correlation filtering method, the tracking accuracy of the tracker based on the deep learning method is greatly improved, but the calculation cost is higher. Benefiting from the continuous improvement of the performance of computer hardware, the target tracking method based on deep learning has developed rapidly in recent years. Among them, the deep learning model based on attention mechanism has a strong global modeling ability and high performance, and has achieved rapid development in multiple computer vision fields including target tracking. The algorithm taking the attention model as the feature extraction network has gradually become the mainstream of the target tracking algorithm.
[0004] The target tracker based on deep learning can be roughly divided into two categories in structure: one is a dual-flow tracker adopting a twin network architecture, and the other is a single-flow tracker. The dual-flow tracker has a pair of twin feature extraction networks, which are respectively used for feature extraction of a target template and a search region, and the target position is obtained by comparing the similarity of the target and the search region. The single-flow tracker mainly relies on a feature extraction network based on an attention model, divides the target template and the search region into a plurality of tokens, and splices them together to be sent into the unique feature extraction network for feature extraction. Compared with the dual-flow tracker, the single-flow tracker can better utilize the global modeling ability of the attention model, so that the target template and the search region have sufficient information interaction in the process of relationship modeling, thereby obtaining better feature expression ability. However, whether it is the dual-flow tracker or the single-flow tracker, the feature extraction network based on attention will equally process all tokens in the search region, which also leads to a relatively large calculation cost of the attention model. SUMMARY
[0005] The purpose of this invention is to provide a dynamic fusion target tracking method that can reduce interference from background information and improve the accuracy and robustness of target tracking.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A dynamic fusion target tracking method includes:
[0008] A single-stream target tracking model is constructed by using an attention-based neural network as a feature extraction network.
[0009] The target template and search region are input into the single-stream target tracking model. The feature extraction network obtains the relationship information between the search region feature map and the fused token. The relationship information of the fused token is used to upsample the search region and restore the search region feature map to the original number of tokens.
[0010] The feature map of the search region, restored to the original number of tokens, is input into the prediction head, and the prediction head outputs the target tracking result.
[0011] The feature extraction network is configured with four attention groups, each consisting of three attention layers. The first, second, and third attention groups predict the target location by using the attention score maps of their respective attention layers. They then perform token fusion on the feature maps of the search region based on the predicted target location and record the relationship information of the fused tokens. The number of tokens in the search regions of the first, second, third, and fourth attention groups decreases proportionally.
[0012] Furthermore, the feature extraction network includes a connected token embedding layer and 12 attention layers, each of which includes a multi-head attention encoder with 12 heads. The target template and search region input to the feature extraction network are projected into tokens by the token embedding layer and then concatenated together as a unified token sequence, which is then input into the attention layer for computation.
[0013] Furthermore, the first, second, and third attention groups each predict the target's location using the attention score maps of their respective attention layers, obtaining the predicted target location as follows:
[0014] The first attention group accumulates the interaction attention score maps of its three attention layers, and selects the token with the highest score from the accumulated attention score map as the prediction target position of the first attention group.
[0015] The second and third attention groups respectively rank the interaction attention score maps of the last attention layer thereof, and select the token with the highest score from the ranking as the predicted target position of the second and third attention groups.
[0016] Further, the first, second and third attention groups respectively perform token fusion on the search region feature map according to the predicted target position thereof, which comprises:
[0017] The first, second and third attention groups respectively perform cluster score calculation on all tokens in the search region according to the predicted target position thereof, to obtain the cluster scores of all tokens in the search region.
[0018] The cluster scores of all tokens in the search region are ranked from high to low, and the top 30% of tokens are selected as cluster center points from the ranking.
[0019] The cluster center points and the tokens with the smallest feature distance thereto are merged.
[0020] Further, the calculation formula of the cluster score is:
[0021]
[0022] wherein s i is the cluster score of the i-th token in the search region, l i is the local density approximation of the i-th token in the search region, k is the number of tokens to be fused, x j is the feature map of the j-th token in the search region, x i is the feature map of the i-th token in the search region, X is the token set in the search region, d i is the distance index of the i-th token in the search region, l j is the local density approximation of the j-th token in the search region, h i , w i are the longitudinal coordinate and the horizontal coordinate of the i-th token in the search region, h c , w c are the longitudinal coordinate and the horizontal coordinate of the token corresponding to the predicted target position of the previous attention group.
[0023] Further, the feature map of the merged token is obtained by information amount weighted merging of the feature maps of the tokens before merging, and the feature map of the merged token is:
[0024]
[0025] wherein x i ′ is the feature map of the merged token, x j is the feature map of the jth token in the search region, a j is the attention score of the jth token in the search region for the central token pair of the target template, and C is the set of tokens participating in the merging.
[0026] Further, the search region is up-sampled using the relationship information of the fused token to restore the search region feature map to the original token number 256, and the token that has been fused is separated into an independent token, and the feature map thereof is the same as before fusion.
[0027] Compared with the prior art, the beneficial effects of the present application are:
[0028] The dynamic fusion target tracking method provided by the present application effectively utilizes the uniqueness of the target in the single-stream target tracking model, uses the intermediate result of the feature extraction network based on the attention mechanism to obtain the prior information of the target position, and uses this prior information to fuse the potential background token away from the target position, so that the number of tokens in the search region of the feature extraction network gradually decreases, while reducing the computational amount of the attention model, the feature granularity of the non-target region in the search region is reduced by merging the background token, the interference of the background information is reduced, and the accuracy and robustness of target tracking are improved. The target position is predicted in stages by ranking the interactive attention, and the predicted target position is used to reduce the pollution of the potential background region to the target feature in the form of token fusion, improve the prediction accuracy, and reduce the number of tokens that need to be calculated by the feature extraction network, and reduce the computational complexity. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 is a flowchart of the dynamic fusion target tracking method provided by the embodiment of the present application. DETAILED DESCRIPTION
[0030] The technical solutions of the present application will be further described in detail below with reference to specific embodiments.
[0031] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application. The technical features in the embodiments of the present application and the embodiments can be combined with each other without conflict.
[0032] Embodiment 1:
[0033] Figure 1 is a dynamic fusion target tracking method flowchart provided by the embodiment. The flowchart only shows the logical sequence of the method of the embodiment. The steps shown or described can be completed in an order different from that shown on the premise that they do not conflict with each other. Figure 1
[0034] Referring to Figure 1 , the method of the embodiment specifically includes the following steps:
[0035] Step 1: using a neural network based on an attention mechanism as a feature extraction network to construct a single-stream target tracking model;
[0036] Step 2: inputting a target template and a search area into the single-stream target tracking model, obtaining search area feature maps and relationship information of fusion tokens by the feature extraction network, and using the relationship information of the fusion tokens to upsample the search area to restore the search area feature maps to the original token number;
[0037] Step 3: inputting the search area feature maps restored to the original token number into a prediction head to output a target tracking result.
[0038] The feature extraction network includes a connected token embedding layer and 12 layers of attention layers, and each layer of attention layer includes a multi-head attention encoder with 12 heads. The target template and the search area input into the feature extraction network are projected into tokens by the token embedding layer and spliced together as a unified token sequence input into the attention layer for operation. The feature extraction network sets four groups of attention groups with every three layers of attention layers as a group. The first, second, and third groups of attention groups respectively predict the target through the attention score maps of the attention layers thereof, obtain the predicted target position, and perform token fusion on the search area feature maps according to the predicted target position to record the relationship information of the fusion tokens. The number of tokens in the search area of the first, second, third, and fourth groups of attention groups decreases proportionally.
[0039] The first, second, and third groups of attention groups respectively predict the target through the attention score maps of the attention layers thereof to obtain the predicted target position, including the following steps:
[0040] Step 1: the first group of attention groups accumulates the interaction attention score maps of the three layers of attention layers thereof, and selects the token with the highest score from the accumulated attention score map as the predicted target position of the first group of attention groups;
[0041] Step 2: The second and third attention groups respectively sort the interaction attention score maps of the last attention layer thereof, and filter out the token with the highest score from the sorting as the predicted target position of the second and third attention groups.
[0042] The token fusion of the first, second and third attention groups on the search region feature map according to the predicted target position thereof respectively includes the following steps:
[0043] Step 1: The first, second and third attention groups respectively perform cluster score calculation on all tokens in the search region according to the predicted target position thereof, to obtain the cluster scores of all tokens in the search region.
[0044] The calculation formula of the cluster score is:
[0045]
[0046] wherein, s i is the cluster score of the i-th token in the search region, l i is the local density approximation of the i-th token in the search region, k is the number of tokens to be fused, x j is the feature map of the j-th token in the search region, x i is the feature map of the i-th token in the search region, X is the token set in the search region, d i is the distance index of the i-th token in the search region, l j is the local density approximation of the j-th token in the search region, h i , w i are the longitudinal coordinate and horizontal coordinate of the i-th token in the search region, h c , w c are the longitudinal coordinate and horizontal coordinate of the token corresponding to the predicted target position of the previous attention group.
[0047] Step 2: Sort all tokens in the search region from high to low according to the cluster scores thereof, and filter out the top 30% of tokens as cluster center points.
[0048] Step 3: Merge the cluster center points and the token with the smallest feature distance thereto.
[0049] The feature map of the merged token is obtained by information amount weighted merging of the feature maps of the tokens before merging, and the feature map of the merged token is:
[0050]
[0051] wherein x i ′ is the feature map of the merged token, x j is the feature map of the jth token in the search region, a j is the attention score of the jth token in the search region to the central token of the target template, represents the information size of the jth token in the search region, and C is the set of tokens participating in the merging.
[0052] In this embodiment, the target template and the search region are obtained from a tracking video sequence. The original input images are all matrices of [C, H, W], where C, H and W respectively correspond to the channel number, height and width of the image. The resolution of the target template image is 3x128x128, and the resolution of the search region image is 3x256x256. After being projected by the token embedding layer, the target template and the search region are respectively divided into 64 tokens and 256 tokens, and the image size corresponding to each token is 16x16.
[0053] In this embodiment, the target position is predicted by relying on the operation results of the multi-head attention encoder. In the operation results output by the multi-head attention encoder, the attention score of each token in the input sequence to all other tokens is included. By using the attention scores of the token at the central position of the target template to all tokens in the search region, these scores are restored to 16x16 interactive attention score maps, which correspond to the token positions of the search region image. The token with the highest attention score is selected as the predicted target position. The interactive attention score maps of the three attention layers of the first group of attention groups are accumulated to improve the accuracy of the first predicted position. The token with the highest score is selected from the accumulated attention score map as the predicted target position of the first group of attention groups. The interactive attention score maps of the last attention layer of the second and third groups of attention groups are sorted respectively, and the token with the highest score is selected from the sorting as the predicted target position of the second and third groups of attention groups.
[0054] In this embodiment, after one token fusion, the number of remaining tokens in the search region is 70% of the original number, and the number of tokens in the search region of the four groups of attention groups is 256, 180, 126 and 88 respectively.
[0055] In this embodiment, the relationship information of the fused tokens is used to upsample the search region, so that the search region feature map is restored to the original token number 256, and the fused tokens are separated into independent tokens again, and their feature maps are the same as before fusion.
[0056] The dynamic fusion target tracking method provided in the embodiment effectively utilizes the uniqueness of the target in the single-stream target tracking model, obtains prior information of the target position by using the intermediate result of the feature extraction network based on the attention mechanism, and fuses potential background tokens away from the target position by using the prior information, so that the number of tokens in the search area of the feature extraction network gradually decreases. While reducing the calculation amount of the attention model, the feature granularity of the non-target area in the search area is reduced by merging the background tokens, the interference of the background information is reduced, the accuracy and robustness of target tracking are improved. The target position is predicted in stages by ranking the interactive attention, and the predicted target position is used to reduce the pollution of the potential background area to the target feature in the token fusion manner, improve the prediction accuracy, and reduce the number of tokens that need to be calculated by the feature extraction network, and reduce the calculation complexity.
[0057] The above is only the preferred embodiment of the present application, and it should be pointed out that for those skilled in the art, without departing from the technical principles of the present application, a number of improvements and modifications can be made, and these improvements and modifications should be considered as the protection scope of the present application.
Claims
1. A dynamic fusion target tracking method, characterized in that, The application relates to a single-stream target tracking model based on an attention mechanism. The target template and the search area are input into the single-stream target tracking model, the search area feature map and the relationship information of the fused token are obtained by the feature extraction network, the search area is up-sampled by using the relationship information of the fused token, and the search area feature map is restored to the original token number. The search area feature map restored to the original token number is input into the prediction head, and the target tracking result is output by the prediction head. The first, second and third attention groups respectively predict the target by using the attention score maps of the attention layers, obtain the predicted target positions, fuse the tokens of the search area feature map according to the predicted target positions, and record the relationship information of the fused tokens; the token numbers in the search areas of the first, second, third and fourth attention groups are decreased at equal ratios. The first, second and third attention groups respectively predict the target by using the attention score maps of the attention layers, obtain the predicted target positions, fuse the tokens of the search area feature map according to the predicted target positions, and record the relationship information of the fused tokens; the token numbers in the search areas of the first, second, third and fourth attention groups are decreased at equal ratios. The first, second and third attention groups respectively predict the target by using the attention score maps of the attention layers, obtain the predicted target positions, fuse the tokens of the search area feature map according to the predicted target positions, and record the relationship information of the fused tokens; the token numbers in the search areas of the first, second, third and fourth attention groups are decreased at equal ratios. The first, second and third attention groups respectively predict the target by using the attention score maps of the attention layers, obtain the predicted target positions, fuse the tokens of the search area feature map according to the predicted target positions, and record the relationship information of the fused tokens; the token numbers in the search areas of the first, second, third and fourth attention groups are decreased at equal ratios. The first, second and third attention groups respectively predict the target by using the attention score maps of the attention layers, obtain the predicted target positions, fuse the tokens of the search area feature map according to the predicted target positions, and record the relationship information of the fused tokens; the token numbers in the search areas of the first, second, third and fourth attention groups are decreased at equal ratios. The feature extraction network comprises a connected token embedding layer and 12 attention layers, each attention layer comprises a multi-head attention encoder with 12 heads; the target template and the search area input into the feature extraction network are projected into tokens by the token embedding layer, spliced together and input into the attention layer as a unified token sequence for operation. The formula for calculating the cluster score is: The feature map of the merged token is obtained by weighting and merging the feature maps of the tokens before merging, and the feature map of the merged token is: The search area is up-sampled by using the relationship information of the fused token, the search area feature map is restored to the original token number 256, and the fused token is separated into an independent token again, and the feature map is the same as before fusion.
2. The dynamic fusion target tracking method of claim 1, wherein, 3. The dynamic fusion target tracking method of claim 1, wherein, ； wherein, is a cluster score of the i-th token in the search region, is a local density approximation of the i-th token in the search region, , is a number of tokens that need to be fused, is a feature map of the i-th token in the search region, is a feature map of the i-th token in the search region, is a set of tokens in the search region, is a distance indicator of the i-th token in the search region, , is a local density approximation of the i-th token in the search region, , is a vertical coordinate, a horizontal coordinate of the i-th token in the search region, , is a vertical coordinate, a horizontal coordinate of a token corresponding to a predicted target position of a previous group of attention. 4. The dynamic fusion target tracking method of claim 1, wherein, ； wherein, is a feature map of the merged token, is a feature map of the token in the search region, is a feature map of the token in the search region, is an attention score of the target template central token pair to the token in the search region, is an attention score of the target template central token pair to the token in the search region, is a set of tokens participating in the merging.
5. The dynamic fusion target tracking method of claim 1, wherein,
Citation Information
Patent Citations
Real-time visual target tracking method and device, and storage medium
CN116403006A