A twin network target tracking algorithm with enhanced positioning quality information
By introducing CNN network and dual-branch area recommendation network into the twin fully convolutional network algorithm, combined with the enhanced classification training method of positioning quality information, the adaptability and robustness of the existing algorithm in complex scenarios is solved, and a more efficient and reliable target tracking effect is achieved.
Patent Information
- Application Number
- CN202210679583.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-15
- Publication Date
- 2025-03-11
- Estimated Expiration
- 2042-06-15
AI Technical Summary
The existing twin fully convolutional network algorithm has poor adaptability when tracking targets in complex scenarios, especially in the case of fuzzy targets, low resolution, scale changes, etc., the robustness and real-timeness are limited.
A twin network target tracking algorithm with enhanced positioning quality information was designed. By introducing CNN network and dual-branch area recommendation network, the target prediction process is optimized, the semantic information in the feature map is fully utilized, and the model's discrimination ability and anti-interference ability are improved through the enhanced classification training method of fused positioning quality information.
The classification and boundary prediction process is improved, the model's ability to describe the target and anti-interference ability of the target scale changes is enhanced, and the tracking speed and credibility of the results are improved.
Smart Images
Figure CN115035452B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of graphics and image processing, and in particular relates to a twin network target tracking algorithm with enhanced positioning quality information. Background Art
[0002] Object tracking requires locating the object in subsequent frames using the limited information provided by the first frame in the video. Object tracking technology has good application potential in fields such as autonomous driving, augmented reality, pedestrian detection and real-time monitoring. At the same time, object tracking also faces many challenges brought by complex situations, such as changing lighting conditions, object deformation, partial occlusion and image blur. Therefore, achieving robust tracking in complex scenes is still a challenging task.
[0003] The fully convolutional twin network algorithm uses a twin-structured convolutional neural network to track targets. Specifically, the algorithm uses a twin neural network to compare the similarity between the target and the search area in the subsequent video frames to obtain a similarity matrix. The maximum value in the similarity matrix is the target position. The similarity matrix is then expanded through bicubic interpolation to obtain the approximate range of the target, and multi-scale testing is used to predict the target boundary to achieve target tracking. However, the algorithm has poor adaptability to problems such as target blur, low resolution, and scale changes, and its application scenarios are greatly limited.
[0004] The algorithm framework of the twin full convolutional network algorithm is as follows Figure 1 The algorithm consists of two parallel branches, with the weight-sharing Alexnet network as the backbone network. The two branches take the selected target template Z and the search area X cropped from the subsequent frame as input respectively. Feature maps of size 6×6×128 and 22×22×128 are generated respectively, and the similarity matrix f(Z,X) of size 17×17×1 is generated using formula (1), where * is the cross-correlation operation and b is the deviation.
[0005]
[0006] After obtaining the similarity matrix, the similarity matrix is expanded to 272×272×1 through bicubic interpolation, and then {-2,-1,0,1,2} The test was performed on 5 scales and the best scale was selected to track the target.
[0007] The twin fully convolutional neural network directly uses classification labels as training labels during the training process, causing the model to learn incorrect background information, reducing the credibility of the similarity matrix. In addition, the similarity matrix generated by the cross-correlation operation is directly used to predict the target position during the prediction process, and the semantic information contained in the target features is not fully utilized, resulting in limited algorithm discrimination. At the same time, the strategy of using multi-scale testing to predict the target boundary makes the algorithm less robust to the scale change of the target. When the target changes in scale or rotates, it cannot stably track the target, and the real-time performance of the algorithm is affected.
[0008] In addition, the model learns background information during training, resulting in reduced classification accuracy; the twin fully convolutional network predicts the target position by interpolating the response map, but cannot fully utilize semantic information, resulting in poor model discrimination. Multi-scale testing results in poor robustness of the algorithm to target scale changes and deformations, and the real-time performance of the algorithm is affected. Summary of the invention
[0009] Based on the above deficiencies in the prior art, the technical problem solved by the present invention is to provide a twin network target tracking algorithm with enhanced positioning quality information, which can improve the classification and boundary prediction process, and introduce a CNN network to enable the model to make more full use of the semantic information in the feature map, and the prediction results have a higher credibility.
[0010] In order to solve the above technical problems, the present invention is implemented by the following technical solutions:
[0011] The twin network target tracking algorithm with enhanced positioning quality information provided by the present invention comprises the following steps:
[0012] Step 1: Determine the target, obtain the first frame image, determine the annotation box using known information, crop the target area, and obtain the target template;
[0013] Step 2: Similarity comparison: Get the current frame, use the feature extraction sub-network to obtain the feature maps of the target and search area, and perform similarity comparison to obtain the similarity matrix;
[0014] Step 3: Target positioning, fusion classification branch performs classification task based on similarity matrix and outputs similarity prediction score;
[0015] Step 4: Boundary prediction: the regression branch predicts the target boundary based on the similarity matrix and selects the best prediction box based on the similarity prediction score.
[0016] Step 5: Output the tracking results, output the target bounding box, and continue with step 2 to track the target of the next frame.
[0017] Furthermore, in step 3, the classification branches are fused to pass ψ clsThe adjusted feature map is used as input and the output similarity prediction matrix of size 17×17×1 is used. The regression branch is trained by ψ reg The adjusted feature map is used as input, the target boundary is predicted based on each positive sample point, and the best prediction result is selected in combination with the similarity prediction matrix.
[0018] Furthermore, in step 4, the loss function used by the regression branch is It is calculated as follows:
[0019]
[0020] In the formula, B and B * They represent the predicted box and the labeled box respectively, C represents the area of the minimum enclosing rectangle of the predicted box and the labeled box, Union(·) represents the union of two rectangles, and IoU * It represents the intersection-over-union ratio of the predicted box and the labeled box.
[0021] From the above, the twin network target tracking algorithm with enhanced positioning quality information of the present invention has the following beneficial effects:
[0022] 1. Design a region recommendation subnetwork, introduce fusion classification and regression branches, use CNN network to optimize the target prediction process, avoid interpolation and multi-scale prediction, and make better use of the semantic information in the feature map.
[0023] 2. In the fusion classification branch, the positioning quality information is introduced to enhance the classification label, and the background information of the edge of the labeled target is downgraded, so that the model pays more attention to the target itself, thereby improving the model's discrimination ability.
[0024] 3. Using an independent regression branch to predict the target boundary avoids the fixed scale limitation brought by multi-scale testing, is more robust to target scale changes, reduces the computational burden, and tracks faster.
[0025] 4. An optimized twin network structure is proposed to improve the classification and boundary prediction process. The introduction of the CNN network enables the model to make better use of the semantic information in the feature map, and the prediction results have a higher credibility.
[0026] 5. An enhanced classification training method that integrates positioning quality information is proposed. The classification labels are weighted using prior spatial scores, so that the model can focus on the target itself during training and tracking, thereby improving the model's ability to describe the target and having stronger anti-interference capabilities.
[0027] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following is a detailed description in combination with the preferred embodiments and the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings of the embodiment are briefly introduced below.
[0029] Figure 1 This is the framework diagram of the twin fully convolutional network algorithm;
[0030] Figure 2 A framework diagram of the tracking algorithm that integrates positioning quality information;
[0031] Figure 3 This is the structure diagram of the head convolutional network;
[0032] Figure 4 This is the effect diagram of fusion classification training;
[0033] Figure 5 This is a flowchart of the twin network target tracking algorithm with enhanced positioning quality information of the present invention. DETAILED DESCRIPTION
[0034] The following is a detailed description of the specific embodiments of the present invention in conjunction with the accompanying drawings, which is a part of this specification and illustrates the principles of the present invention through examples. Other aspects, features and advantages of the present invention will become clear from the detailed description. In the referenced drawings, the same or similar components in different figures are represented by the same reference numerals.
[0035] like Figures 1 to 5 As shown, the steps of the twin network target tracking algorithm with enhanced positioning quality information of the present invention are as follows:
[0036] 1. Determine the target. Get the first frame of the image, use the known information to determine the annotation box, crop the target area, and obtain the target template.
[0037] 2. Similarity comparison. Get the current frame, use the feature extraction subnetwork to obtain the feature maps of the target and search area and perform similarity comparison to obtain the similarity matrix f i (Z,X).
[0038] 3. Target positioning. The fusion classification branch is based on the similarity matrix f cls (Z,X) performs classification tasks and outputs similarity prediction scores.
[0039] 4. Boundary prediction. The regression branch is based on the similarity matrix f reg(Z, X) predicts the target boundary and selects the best prediction box based on the similarity prediction score.
[0040] 5. Output the tracking result. Output the target bounding box and continue to step 2 to track the target in the next frame.
[0041] This paper takes the twin fully convolutional network as the basic framework, introduces a dual-branch region recommendation network containing a classification prediction branch and a boundary prediction branch, and proposes a twin network target tracking algorithm with enhanced positioning quality information. Figure 2 shown.
[0042] The algorithm structure is mainly divided into two parts, the feature extraction subnetwork on the left and the region recommendation subnetwork on the right. The feature extraction subnetwork is a twin network consisting of a template branch Z and a search region branch X. It takes the target in the first frame and the search region in the subsequent frames as input and uses formula (2) to obtain the similarity matrix f i (Z,X).
[0043]
[0044] In the formula, ψ i represents the convolutional layer that adjusts the size of the feature map, i∈{cls,reg} represents different subtasks, cls represents the classification subtask, reg represents the boundary regression subtask, represents the backbone network, * represents the cross-correlation operation, and b represents the bias.
[0045] The region recommendation subnetwork consists of two branches: fusion classification and boundary regression. The two branches have the same structure and use lightweight convolutional neural networks as the head to perform classification and regression tasks respectively. The convolutional network structure is as follows: Figure 3 As shown in the figure, Conv represents the convolution layer, Kernel size represents the convolution kernel size, s represents the step size, k represents the number of convolution kernels, and ReLU represents the activation function.
[0046] The classification branch is integrated through ψ cls The adjusted feature map is used as input and the output similarity prediction matrix of size 17×17×1 is used. The regression branch is trained by ψ reg The adjusted feature map is used as input, the target boundary is predicted based on each positive sample point, and the best prediction result is selected in combination with the similarity prediction matrix.
[0047] In the fusion classification branch, the prior space score is used as the positioning quality score to weight the target and form a positioning quality label to improve the classification training process. ss The calculation method is shown in formula (3).
[0048]
[0049] In the formula, l * ,r * ,t * ,b * Respectively represent the offsets of the annotation point to the left, right, top, and bottom sides of the target annotation box. ss In the scoring, the closer the point is to the center, the higher its score is, while the farther the point is from the center, the lower its score is. The effect of fusion classification training weighted by positioning quality information is as follows: Figure 4 shown.
[0050] The loss function used by the fusion classification branch is shown in formula (4):
[0051]
[0052] In the formula, α t and p t The calculation methods of are shown in formula (5) and formula (6) respectively:
[0053]
[0054]
[0055] In the formula, α is a hyperparameter and p represents the similarity prediction result.
[0056] The loss function used by the regression branch is The calculation method is shown in formula (7):
[0057]
[0058] In the formula, B and B * They represent the predicted box and the labeled box respectively, C represents the area of the minimum enclosing rectangle of the predicted box and the labeled box, Union(·) represents the union of two rectangles, and IoU * It represents the intersection-over-union ratio of the predicted box and the labeled box.
[0059] The training objective is shown in formula (8):
[0060]
[0061] Where N pos represents the number of positive samples, 1 {·} is the indicator function, when 1 if the value is 0, otherwise it is 0.
[0062] The present invention proposes an optimized twin network structure to improve the classification and boundary prediction process. The introduction of the CNN network enables the model to make fuller use of the semantic information in the feature map, and the prediction results have a higher degree of credibility. An enhanced classification training method integrating positioning quality information is proposed. The classification labels are weighted by the prior space score, so that the model focuses on the target itself during training and tracking, which improves the model's ability to describe the target and has stronger anti-interference ability.
[0063] The above is only a preferred embodiment of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and changes can be made without departing from the principle of the present invention, and these improvements and changes are also regarded as the protection scope of the present invention.
Claims
1. A twin network target tracking algorithm with enhanced positioning quality information, characterized in that: The following steps are involved: Step 1: Determine the target, obtain the first frame image, determine the annotation box using known information, crop the target area, and obtain the target template; Step 2: Similarity comparison: Get the current frame, use the feature extraction sub-network to obtain the feature maps of the target and search area, and perform similarity comparison to obtain the similarity matrix; Step 3: Target positioning, fusion classification branch performs classification task based on similarity matrix and outputs similarity prediction score; Step 4: Boundary prediction: the regression branch predicts the target boundary based on the similarity matrix and selects the best prediction box based on the similarity prediction score. Step 5: Output the tracking result, output the target bounding box, and continue to step 2 to track the target in the next frame; The algorithm structure is mainly divided into two parts: the feature extraction subnetwork on the left and the region recommendation subnetwork on the right. The feature extraction subnetwork is a twin network consisting of a template branch Z and a search region branch X. It takes the target in the first frame and the search region in the subsequent frames as input and uses formula (2) to obtain the similarity matrix f i (Z,X); In the formula, ψ i represents the convolutional layer that adjusts the size of the feature map, i∈{cls,reg} represents different subtasks, cls represents the classification subtask, reg represents the boundary regression subtask, represents the backbone network, * represents the cross-correlation operation, and b represents the bias; The classification branch is integrated through ψ cls The adjusted feature map is used as input and the output similarity prediction matrix of size 17×17×1 is used. The regression branch is trained by ψ reg The adjusted feature map is used as input, the target boundary is predicted based on each positive sample point, and the best prediction result is selected in combination with the similarity prediction matrix; In the fusion classification branch, the prior space score is used as the positioning quality score to weight the target and form a positioning quality label to improve the classification training process. ss The calculation method is shown in formula (3); In the formula, l * ,r * ,t * ,b * Respectively represent the offsets of the annotation point to the left, right, top, and bottom sides of the target annotation box; ss In the scoring, the closer the point is to the center, the higher its score is, while the farther away from the center, the lower its score is; The loss function used by the fusion classification branch is shown in formula (4): In the formula, α t and p t The calculation methods of are shown in formula (5) and formula (6) respectively: In the formula, α is a hyperparameter, and p represents the similarity prediction result; The loss function used by the regression branch is The calculation method is shown in formula (7): In the formula, B and B * They represent the predicted box and the labeled box respectively, C represents the area of the minimum enclosing rectangle of the predicted box and the labeled box, Union(·) represents the union of two rectangles, and IoU * Represents the intersection-over-union ratio of the predicted box and the labeled box; The training objective is shown in formula (8): Where N pos represents the number of positive samples, 1 {·} is the indicator function, when 1 if the value is 0, otherwise it is 0.