Ranking-based twin network object tracking method and system
By constructing classification ranking and IoU ranking loss functions and connecting the classification and regression branches of the Siamese network, the failure problem caused by difficult negative samples in Siamese network target tracking is solved, thus improving tracking accuracy and stability.
Patent Information
- Application Number
- CN202210549797.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-20
- Publication Date
- 2026-02-03
- Estimated Expiration
- 2042-05-20
AI Technical Summary
Existing twin network target tracking algorithms are prone to failure when dealing with difficult negative samples, and the independent processing of classification and regression tasks leads to insufficient accuracy.
We construct a classification ranking loss function and an IoU-based ranking loss function, connect the classification and regression branches, and guide the training of the Siamese RPN network through the total loss function, constraining the classification score of positive samples to be higher than that of difficult negative samples, thereby improving tracking accuracy.
This effectively avoids the tracker erroneously selecting negative samples, improves the accuracy and stability of target tracking in Siamese networks, and enhances the synergistic effect of classification and regression.
Smart Images
Figure CN114926500B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of pattern recognition and computer vision, specifically to a ranking-based Siamese network target tracking method and system. Background Technology
[0002] Recently, the target tracking algorithms SiamRPN and SiamRPN++ based on Siam networks have attracted considerable attention. Currently, Siam network-based trackers primarily describe visual tracking as two independent subtasks: classification and regression. When learning the classification sub-network, these methods process each sample separately; for example, if a sample is labeled as positive (1) or negative (0), the network, guided by the classification loss function, tries to output a classification score of 1 or 0 for that sample, but the relationship between positive and negative samples is not considered. This makes it difficult for Siam networks to distinguish hard negative samples (objects similar to the tracked target). During the training phase, the classification sub-network is responsible for classifying the training samples, i.e., simple negative samples containing a large number of semantically uninformative negative samples. Meanwhile, some hard negative samples, due to their extremely small number, are easily overwhelmed by the large number of simple negative samples during the training phase. During testing, although most non-target samples (falling on the background area) can be identified as background by the classifier, as long as there is a distracting target with a high foreground classification score, it can interfere with the tracker. Once its score exceeds the classification score of the real target in a certain frame, the tracker will bias towards the distracting target, causing tracking failure. This has occurred frequently in previous Siamese network trackers.
[0003] Furthermore, a mismatch exists between classification and regression because these tasks are handled independently. Specifically, the classification loss function prompts the model to distinguish between foreground and background, without considering the regression branch. The regression branch, however, aims to regress the target's bounding box for all positive samples without considering the sample's classification. Therefore, samples with high target box regression accuracy may have relatively low target classification scores, while samples with high target classification scores may produce lower regression accuracy. Thus, improving the target tracking accuracy of Siamese RPN networks becomes a pressing issue. Summary of the Invention
[0004] To address the aforementioned technical problems, this invention provides a ranking-based twin network target tracking method and system.
[0005] The technical solution of this invention is: a ranking-based twin network target tracking method, comprising:
[0006] Step S1: Construct a classification ranking loss function and train the classification branch in the Siamese RPN target tracking network;
[0007] Step S2: Construct an IoU-based ranking loss function to align the classification and regression branches in the Siamese RPN target tracking network;
[0008] Step S3: Combine the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct the total loss function, which guides the training of the Siamese RPN target tracking network.
[0009] Compared with the prior art, the present invention has the following advantages:
[0010] 1. This invention discloses a ranking-based Siamese network target tracking method, which uses a classification ranking loss function to constrain the classification score of positive samples to be greater than the score of difficult negative samples. This allows difficult negative samples to be classified as foreground targets, avoiding the tracker from incorrectly selecting negative samples and causing tracking failure.
[0011] 2. This invention utilizes an IoU-based ranking loss function to connect the classification and regression branches in a Siamese RPN network, so that the scores predicted by the classification branch can reflect the classification accuracy and regression prediction accuracy. Attached Figure Description
[0012] Figure 1 This is a flowchart of a ranking-based twin network target tracking method according to an embodiment of the present invention;
[0013] Figure 2 This is a system architecture diagram of the twin RPN target tracking network in an embodiment of the present invention;
[0014] Figure 3 This is a structural block diagram of a ranking-based twin network target tracking system according to an embodiment of the present invention. Detailed Implementation
[0015] This invention provides a ranking-based Siamese network target tracking method. By constructing a classification ranking loss and an IoU-based ranking loss function, the target tracking accuracy of existing Siamese networks can be effectively improved.
[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.
[0017] Example 1
[0018] like Figure 1 As shown in the figure, an embodiment of the present invention provides a ranking-based twin network target tracking method, which includes the following steps:
[0019] Step S1: Construct a classification ranking loss function and train the classification branch in the Siamese RPN target tracking network;
[0020] Step S2: Construct an IoU-based ranking loss function to align the classification and regression branches in the Siamese RPN target tracking network;
[0021] Step S3: Combine the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct the total loss function, which guides the training of the Siamese RPN target tracking network.
[0022] like Figure 2 The diagram illustrates the system architecture of the Siamese RPN target tracking network. It has two inputs: a template image containing the target in the first frame and a search image containing the target and background in subsequent frames. The backbone network extracts features from both images. The two feature maps are input to the RPN module, where they are first fused. The fused features are then input into a classification branch to predict the category of each sample (candidate box), i.e., whether it belongs to the foreground or background; and a regression branch to predict the bounding box of the target.
[0023] In one embodiment, step S1: Constructing a classification ranking loss function and training the classification branch in the Siamese RPN target tracking network, specifically includes:
[0024] Step S11: Calculate the mean foreground classification scores of the positive and negative samples output by the classification branches in the Siamese RPN network according to the following formula (1):
[0025]
[0026] Among them, A pos It is a positive sample set, A neg It is a difficult negative sample set; negative sample j - The weighting coefficient is exp() is an exponential function; positive sample j + weight w j+ for Where N pos It is the positive sample set A pos Number of samples; p j+ and p j- These are the positive samples j predicted by the classification branch. + and difficult negative sample j - Foreground classification score;
[0027] In this embodiment of the invention, based on the foreground classification score output by the classification branch in the Siamese RPN network, all negative samples are sorted, and negative samples with a foreground classification score below 0.5 are filtered out, leaving the remaining negative samples to form a hard negative sample set A. neg ;
[0028] Step S12: Based on P obtained in step S11 + and P - The classification ranking loss function is constructed as shown in formula (2):
[0029]
[0030] Where exp() and log() are the exponential and logarithmic functions, respectively; β is a parameter controlling the magnitude of the loss value, and α is a parameter controlling the ranking distance. In this embodiment of the invention, β can be 4 and α can be 0.5. Formula (2) can constrain the mean P of the positive sample foreground classification score. + Greater than the mean P-value of the difficult negative sample foreground classification - Specifically, if P - P + If the value is large, then L rank_cls The loss value will be large; conversely, if P - P + If the value is small, then L rank_cls The loss value will be smaller. Therefore, during backpropagation, the neural network, in order to make L... rank_cls Making the value as small as possible will constrain P. - Try to be more than P + The low score effectively reduces the classification score of difficult negative samples, thus achieving the goal of suppressing difficult backgrounds.
[0031] This invention transforms the classification problem in Siamese RPN networks into a ranking problem, constraining the positive foreground classification score to be greater than the difficult negative foreground classification score. Compared to existing classification problems, the ranking method provided by this invention acts as a relaxed constraint, where the mean P of the difficult negative foreground classification in formula (2) is... - The values can be very large and fall into the foreground object category, but as long as their foreground classification scores are kept within the mean P, they are acceptable. - Foreground classification score P lower than the mean of positive samples + This prevents the tracker from incorrectly selecting negative samples, which could lead to tracking failure.
[0032] In one embodiment, step S2 above, constructing an IoU-based ranking loss function to align the classification and regression branches in the Siamese RPN target tracking network, specifically includes:
[0033] Step S21: For positive sample i + ,j + ∈A pos In this regard, the following constraint method is agreed upon, as shown in formula (3):
[0034]
[0035] in, and They are positive samples i + and j + Foreground classification score; and They are positive samples i + and j + The regression score predicted by the regression branch is represented by IoU (Intersection over Union);
[0036] According to formula (3), when the positive sample i + regression score Greater than positive sample j + regression score When, then the constrained sample i + Foreground classification score Greater than positive sample j + Foreground classification score Similarly, when positive sample i + Foreground classification score Greater than positive sample j + Foreground classification score Then, positive sample i can be constrained. + regression score Greater than positive sample j + regression score By applying the constraints of formula (3), the score of the classification branch can reflect not only the foreground classification accuracy of the target, but also the regression accuracy of the target.
[0037] Step S22: Construct the IoU-based ranking loss function L rank-iou As shown in the following formula (4):
[0038]
[0039] Where xp() is an exponential function; γ is a parameter that controls the magnitude of the loss value. In this embodiment of the invention, γ can be 3.
[0040] The first term of formula (4) reflects the first constraint in formula (3), that is, when Greater than hour, Greater than The smaller the value of the first term in formula (4), the better; similarly, the second term in formula (4) reflects the second constraint in formula (3), that is, when Greater than hour, Greater than The smaller the value of the second term in formula (4), the higher the classification score of the sample with higher regression accuracy will be under the action of formula (4). The classification score of the classification branch can reflect the accuracy of the target box prediction to a certain extent. In this way, formula (4) can connect the classification branch and the regression branch, and the classification branch can reflect the classification and regression accuracy at the same time.
[0041] This invention constructs an IoU-based ranking loss function that can connect the classification branch and the regression branch without adding extra branches, allowing the classification branch to simultaneously reflect the classification accuracy and regression accuracy (i.e., the accuracy of the bounding box prediction).
[0042] In one embodiment, step S3 above—combining the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct a total loss function to guide the training of the Siamese RPN target tracking network—specifically includes:
[0043] The classification ranking loss function L rank_cls The ranking loss function L based on IoU rank_iou And the original loss function L in the RPN network RPN Add them together to construct the total loss function L. total As shown in the following formula (5):
[0044] L total =L RPN +L rank-cls +L rank-iou (5)
[0045] Through L total It can guide the training of the twin RPN target tracking network, and be used to optimize and update network parameters.
[0046] Example 2
[0047] like Figure 3 As shown, this embodiment of the invention provides a ranking-based Siamese network target tracking system, including the following modules:
[0048] A classification ranking loss function module 41 is constructed to build the classification ranking loss function and train the classification branch in the Siamese RPN target tracking network.
[0049] A module 42 for constructing an IoU-based ranking loss function is provided to align the classification and regression branches in the Siamese RPN target tracking network.
[0050] The module 43 for constructing the total loss function combines the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct the total loss function, which guides the training of the Siamese RPN target tracking network.
[0051] The above embodiments are provided merely for the purpose of describing the present invention and are not intended to limit the scope of the invention. The scope of the invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the invention should be covered within the scope of the invention.
Claims
1. A ranking-based Siamese network target tracking method, characterized in that, include: Step S1: Construct a twin RPN target tracking network, which includes two inputs: a template image containing the target to be tracked in the first frame and a search image containing the target and background in subsequent frames. The backbone network is used to extract features from the two images. The two feature maps are input into the RPN module. First, the two features are fused. The fused features are then input into the classification branch to predict the category of each sample, i.e., whether it belongs to the foreground target or the background. And a regression branch, used to predict the bounding box of the tracked target; Constructing a classification ranking loss function and training the classification branch in the Siamese RPN target tracking network, specifically including: Step S11: Calculate the mean foreground classification scores of the positive and negative samples output by the classification branches in the Siamese RPN network according to the following formula (1): (1) in, It is a positive sample set. It is a difficult negative sample set; negative samples The weighting coefficient is exp() is an exponential function; positive samples weight for ,in It is a positive sample set Number of samples; and These are the positive samples predicted by the classification branches. and difficult negative samples Foreground classification score; Step S12: Based on the results obtained in step S11 and The classification ranking loss function is constructed as shown in formula (2): (2) Among them, exp() and log() are the exponential function and the logarithmic function, respectively; Parameters for controlling the magnitude of the loss value, A parameter to control the sorting distance; Step S2: Construct an IoU-based ranking loss function to align the classification and regression branches in the Siamese RPN target tracking network, specifically including: Step S21: For positive samples In this regard, the following constraint method is agreed upon, as shown in formula (3): (3) in, and They are positive samples and Foreground classification score; and They are positive samples and The regression score predicted by the regression branch; Step S22: Construct an IoU-based ranking loss function As shown in the following formula (4): (4) in, The parameter exp() is an exponential function used to control the magnitude of the loss value. Step S3: Combine the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct the total loss function, which guides the training of the Siamese RPN target tracking network.
2. The ranking-based Siamese network target tracking method according to claim 1, characterized in that, Step S3: Combining the classification ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct a total loss function to guide the training of the Siamese RPN target tracking network, specifically including: The classification and ranking loss function The IoU-based ranking loss function And the original loss function in the RPN network Add them together to construct the total loss function. As shown in the following formula (5): (5)。 3. A ranking-based Siamese network target tracking system, characterized in that, Includes the following modules: A classification and ranking loss function module is constructed to build the Siamese RPN target tracking network. It includes two inputs: a template image containing the target to be tracked in the first frame and a search image containing the target and background in subsequent frames. The backbone network is used to extract features from the two images. The two feature maps are input into the RPN module. First, the two features are fused. The fused features are then input into the classification branch to predict the category of each sample, i.e., whether it belongs to the foreground target or the background. And a regression branch, used to predict the bounding box of the tracked target; Constructing a classification ranking loss function and training the classification branch in the Siamese RPN target tracking network, specifically including: Step S11: Calculate the mean foreground classification scores of the positive and negative samples output by the classification branches in the Siamese RPN network according to the following formula (1): (1) in, It is a positive sample set. It is a difficult negative sample set; negative samples The weighting coefficient is exp() is an exponential function; positive samples weight for ,in It is a positive sample set Number of samples; and These are the positive samples predicted by the classification branches. and difficult negative samples Foreground classification score; Step S12: Based on the results obtained in step S11 and The classification ranking loss function is constructed as shown in formula (2): (2) Among them, exp() and log() are the exponential function and the logarithmic function, respectively; Parameters for controlling the magnitude of the loss value, A parameter to control the sorting distance; A module for constructing an IoU-based ranking loss function is included. This module aligns the classification and regression branches in the Siamese RPN target tracking network. Specifically, it includes: Step S21: For positive samples In this regard, the following constraint method is agreed upon, as shown in formula (3): (3) in, and They are positive samples and Foreground classification score; and They are positive samples and The regression score predicted by the regression branch; Step S22: Construct an IoU-based ranking loss function As shown in the following formula (4): (4) in, The parameter exp() is an exponential function used to control the magnitude of the loss value. A module for constructing a total loss function is used to combine the classification and ranking loss function, the IoU-based ranking loss function, and the original loss function in the RPN network to construct a total loss function to guide the training of the Siamese RPN target tracking network.