Single target tracking method based on cross-layer feature fusion and sorting loss
By introducing cross-layer feature fusion and sorting loss methods in the target tracker, the problems of insufficient robustness and easy interference in the prior art are solved, and more efficient target tracking is achieved, especially in diversified challenge scenarios.
Patent Information
- Application Number
- CN202411763673.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-28
AI Technical Summary
The existing twin network-based target tracker is not robust enough when facing occlusion, scale changes, background chaos, rapid movement, lighting changes and appearance changes, and is easily deceived by interference items, resulting in tracking failure.
A single-objective tracking method based on cross-layer feature fusion and sorting losses is adopted. The cross-layer feature fusion module learns the correlation between the target image and the search image, improves network robustness, and introduces classification sorting loss function and IoU-guided sorting loss function to enhance recognition capabilities and alleviate classification and positioning mismatch problems.
It improves the robustness of the target tracker, can more effectively resist interference items, maintain stable tracking of the target, and significantly improves the tracking accuracy and success rate in diverse challenges scenarios.
Smart Images
Figure CN119723263A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of target tracking technology, and in particular to a single target tracking method based on cross-layer feature fusion and sorting loss. Background Art
[0002] Object tracking is a challenging task in computer vision. Given the state of an object in an initial frame of a video, the tracker needs to predict the state of the object in subsequent frames. Research on object tracking has made great progress in recent years, but object tracking still faces challenges such as occlusion, scale changes, background clutter, fast motion, illumination changes, and appearance changes. Stable and real-time visual trackers have many practical applications, such as augmented reality, autonomous robots, and self-driving cars.
[0003] The key to the tracker based on the twin network is to calculate the correlation between the template branch and the search branch, and obtain the response map to predict the location of the target. This response map is usually calculated by the single-level features extracted by the backbone network. However, since the target may experience large deformation or interference, the correlation between the single-level features will deteriorate, resulting in tracking failure. In the training phase, there are a large number of simple samples in the training samples that can be easily classified, while a small number of interference samples have little effect on network optimization; in the test phase, background interference items may seriously mislead the tracker when they have strong positive confidence, resulting in tracking failure. In addition, the classification and localization tasks of the tracker based on the twin network are handled separately. The classification branch aims to distinguish the target from the background and does not care about the location information of the target; the regression branch locates the bounding box of the target for each positive sample without considering the classification information. Therefore, a well-localized area may have a low foreground confidence, while an area with a high foreground confidence may produce a low positioning accuracy. Summary of the invention
[0004] According to the technical problems raised above, a single target tracking method based on cross-layer feature fusion and ranking loss is provided. The present invention introduces a cross-layer feature fusion module and a ranking loss function. In a deep neural network, shallow features mainly extract low-level features such as color and shape, and deep features can provide rich semantic information. The cross-layer feature fusion module uses the features extracted from different layers of the backbone network to learn the correlation between the target image and the search image, which can improve the robustness of the overall network. The classification ranking loss function enhances the recognition ability by modeling the relationship between foreground samples and background samples, which can prevent the tracker from being deceived by interference items; the IoU-guided ranking loss function connects the two subtasks by aligning the classification scores with the relevant IoU, ensuring that good predictions can be represented by high classification confidence scores, which can effectively alleviate the classification and positioning mismatch problem.
[0005] The technical means adopted by the present invention are as follows:
[0006] A single target tracking method based on cross-layer feature fusion and ranking loss, comprising:
[0007] S1. Obtain a template image and a search image, and use a twin network to extract features from the template image and the search image to obtain template features and search features;
[0008] S2, using the cross-layer feature fusion module, calculate the correlation between the template features and the search features to obtain the response map;
[0009] S3. Send the obtained response map to the classification-regression subnetwork, where the classification branch is used to predict the category of each position and the regression branch is used to calculate the target bounding box of the position;
[0010] S4, introduce the sorting loss function and calculate the loss;
[0011] S5. Use the public datasets COCO, DET, VID, YouTube-BB, GOT-10K training set and LaSOT training set to train the model;
[0012] S6. Test the model performance on the public datasets OTB-100, UAV123, LaSOT test set, and GOT-10K test set to predict the bounding box of the target.
[0013] Furthermore, step S1 specifically includes:
[0014] S11. The improved ResNet-50 is used as the backbone network for feature extraction. Only the first four convolution blocks of ResNet-50 are retained. In order to retain detailed spatial information and increase the receptive field, all 3×3 convolution kernels in the fourth convolution block are expanded at a rate of 2 times, and the convolution step size is reduced from 2 to 1.
[0015] S12, resize the template image and the search image to a specific size, encode them into feature maps through ResNet-50 with the same parameters, and then reduce the number of feature channels to 256 through the neck network;
[0016] S13, extract the template feature z and search feature x from the specific convolution layer, and define the template feature z as f z convi , define the search feature x as f x convi , where i=1,…,4.
[0017] Furthermore, step S2 specifically includes:
[0018] S21. Define single-layer feature correlation as the correlation between two features at the same level, and define double-layer feature correlation as the correlation between features at two different levels.
[0019] S22, using the cross-layer feature fusion module to calculate f z conv3 and f x conv4 The correlation between z conv4 and f x conv3 The correlation between z conv4 and f x conv4 The specific process is described as follows:
[0020]
[0021] in, and They represent the feature response maps of the regression branch and the classification branch respectively, (i,j)∈[(3,4),(4,3),(4,4)], Φ represents a 3×3 convolutional layer, represents the cross-correlation operation;
[0022] S23. According to the trained weights, the feature maps in the two branches are aggregated separately, and the output is sent to the classification-regression sub-network.
[0023] Furthermore, in step S22, the cross-layer correlation feature fusion module adopted is composed of three sub-modules with the same network architecture, which are used to calculate two double-layer feature correlation maps and one single-layer feature correlation map; each sub-module first projects the features to the classification and regression branches through parallel convolutional layers, and then uses deep cross-correlation to generate a response map representing the correlation between the inputs.
[0024] Furthermore, step S3 specifically includes:
[0025] S31, the classification branch consists of five 3×3 convolutional layers, generating a |P cls | w×h×1 The single-channel classification score map represented by is used to calculate the position correspondence between the search image S and the prediction map P as follows:
[0026]
[0027] Among them, (S x ,S y ) and (P x ,P y ) represent the coordinates of the sampled pixels in the search image and the prediction image, respectively, (sw ,s h ) and (w,h) represent the width and height of the search image and prediction image respectively, and s represents the step size;
[0028] S32, the regression branch directly outputs the distance from the pixel to the four sides of the target bounding box, using |P regw×h×1 Represents the regression map of the four channels and calculates the bounding box. The formula is as follows:
[0029] S x0 =S x -l,S y0 =S y -t
[0030]
[0031] Among them, O = (l, t, r, b) represents the offset of the pixel to the left, top, right and bottom boundaries of the target, and the coordinates of the upper left corner and lower right corner represent the rectangular bounding box.
[0032] Further, step S4 specifically includes:
[0033] S41. Introduce the classification ranking loss function and calculate the classification loss. The process is as follows:
[0034] S411. Train a classifier supervised by cross entropy loss.
[0035] S412, sort all negative samples according to the predicted confidence scores, and sort the negative samples with confidence scores lower than τ reg The negative samples are filtered out, and the remaining negative samples constitute the difficult negative sample set where n - represents the number of negative samples, p j- Represents negative sample j - Confidence score of
[0036] S413. For positive samples, retain all positive samples to obtain a positive sample set
[0037] S414. Calculate the expectation of difficult negative samples and positive samples. The calculation formula is as follows:
[0038]
[0039] in, and Represent the weights of negative samples and positive samples respectively;
[0040] S415. Use the cross entropy loss function to calculate the classification loss as follows:
[0041]
[0042] Among them, α and β represent hyperparameters;
[0043] S42, introduce the IoU-guided ranking loss function and calculate the classification loss. The process is as follows:
[0044] S421, for positive sample i,j∈A pos , set the sorting constraints as follows:
[0045]
[0046] Among them, p i and p j Represent the confidence scores of positive samples i and j respectively; Respectively represent the IoU values of positive samples i and j;
[0047] S422. Define the IoU-guided ranking loss function as:
[0048]
[0049] Among them, γ represents the hyperparameter that controls the loss.
[0050] Furthermore, step S5 specifically includes:
[0051] S51. The entire network is trained end-to-end on large-scale datasets, including COCO, DET, VID, YouTube-BB, GOT-10K and LaSOT. The template image size is 127×127 and the search image size is 255×255.
[0052] S52, initialize the backbone network using weights pre-trained on ImageNet;
[0053] S53, the network is trained using stochastic gradient descent with a batch size of 32 and 20 training rounds.
[0054] Further, step S6 specifically includes:
[0055] S61, in the testing phase, cutting out a template image from the first frame, inputting it into a feature extraction network, and caching the extracted template features;
[0056] S62, for subsequent frames, crop the image and extract features according to the target position of the previous frame, and then make predictions in the search area to obtain a classification score map and a regression map;
[0057] S63. After generating the prediction box, use the cosine window and scale change penalty to smooth the movement and change of the target, then select the prediction box with the highest score and update its size by linear interpolation with the state of the previous frame.
[0058] Compared with the prior art, the present invention has the following advantages:
[0059] 1. The present invention provides a single target tracking method based on cross-layer feature fusion and ranking loss. Its cross-layer feature fusion module uses features extracted from different layers of the backbone network to learn the correlation between the target image and the search image, which can improve the robustness of the overall network.
[0060] 2. The present invention provides a single target tracking method based on cross-layer feature fusion and ranking loss, which introduces a ranking loss function and calculates the loss. Its classification ranking loss function enhances the recognition ability by modeling the relationship between foreground samples and background samples, which can prevent the tracker from being deceived by interference items. Its IoU-guided ranking loss function aligns the classification score with the relevant IoU to connect the two subtasks, ensuring that a good prediction can be represented by a high classification confidence score, which can effectively alleviate the classification and positioning mismatch problem.
[0061] Based on the above reasons, the present invention can be widely promoted in the fields of target tracking and the like. BRIEF DESCRIPTION OF THE DRAWINGS
[0062] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0063] Figure 1 The network framework diagram of the method of the present invention.
[0064] Figure 2 A graph of success rate and accuracy on the OTB100 dataset provided by an embodiment of the present invention.
[0065] Figure 3 A diagram of success rate and accuracy on the LaSOT dataset provided by an embodiment of the present invention.
[0066] Figure 4 A graph of success rate and accuracy on the UAV123 dataset provided by an embodiment of the present invention.
[0067] Figure 5 This is a diagram of ablation experiment results provided by an embodiment of the present invention.
[0068] Figure 6A diagram of success rate and accuracy under scale changes provided by an embodiment of the present invention.
[0069] Figure 7 A diagram of the success rate and accuracy under occlusion provided by an embodiment of the present invention.
[0070] Figure 8 A graph of success rate and accuracy under target deformation provided by an embodiment of the present invention.
[0071] Fig. 9 Success rate and accuracy diagram under motion blur provided by an embodiment of the present invention.
[0072] Fig.10 A diagram showing the success rate and accuracy under fast movement provided by an embodiment of the present invention.
[0073] Fig.11 A diagram showing the success rate and accuracy of an embodiment of the present invention under in-plane rotation.
[0074] Fig.12 A graph showing the success rate and accuracy of an embodiment of the present invention when the image is out of view.
[0075] Fig.13 A success rate and accuracy diagram under background clutter provided by an embodiment of the present invention.
[0076] Fig.14 Success rate and accuracy diagram at low resolution provided for embodiments of the present invention.
[0077] Fig.15 A graph showing the success rate and accuracy under varying illumination provided by an embodiment of the present invention.
[0078] Fig.16 Success rate and accuracy diagram for out-of-plane rotation provided by an embodiment of the present invention.
[0079] Fig.17 This is a qualitative analysis result diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0080] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0081] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0082] like Figure 1 As shown in FIG. 1 , it is a network framework diagram of the method of the present invention. The present invention provides a single target tracking method based on cross-layer feature fusion and sorting loss, including:
[0083] S1. Obtain a template image and a search image, and use a twin network to extract features from the template image and the search image to obtain template features and search features;
[0084] S2, using the cross-layer feature fusion module, calculate the correlation between the template features and the search features to obtain the response map;
[0085] S3. Send the obtained response map to the classification-regression subnetwork, where the classification branch is used to predict the category of each position and the regression branch is used to calculate the target bounding box of the position;
[0086] S4, introduce the sorting loss function and calculate the loss;
[0087] S5. Use the public datasets COCO, DET, VID, YouTube-BB, GOT-10K training set and LaSOT training set to train the model;
[0088] S6. Test the model performance on the public datasets OTB-100, UAV123, LaSOT test set, and GOT-10K test set to predict the bounding box of the target.
[0089] In specific implementation, as a preferred embodiment of the present invention, step S1 specifically includes:
[0090] S11. The improved ResNet-50 is used as the backbone network for feature extraction. Only the first four convolution blocks of ResNet-50 are retained. In order to retain detailed spatial information and increase the receptive field, all 3×3 convolution kernels in the fourth convolution block are expanded at a rate of 2 times, and the convolution step size is reduced from 2 to 1.
[0091] S12. In order to reduce the computational cost, the template image and the search image are resized to a specific size and encoded into feature maps by ResNet-50 with the same parameters, and the number of feature channels is reduced to 256 by the neck network;
[0092] S13, extract the template feature z and search feature x from the specific convolution layer, and define the template feature z as f z convi , define the search feature x as f x convi , where i=1,…,4.
[0093] In specific implementation, as a preferred embodiment of the present invention, step S2 specifically includes:
[0094] S21. Define single-layer feature correlation as the correlation between two features at the same level, and define double-layer feature correlation as the correlation between features at two different levels.
[0095] S22, using the cross-layer feature fusion module to calculate f z conv3 and f x conv4 The correlation between z conv4 and f x conv3 The correlation between z conv4 and f x conv4 The specific process is described as follows:
[0096]
[0097] in, and They represent the feature response maps of the regression branch and the classification branch respectively, (i,j)∈[(3,4),(4,3),(4,4)], Φ represents a 3×3 convolutional layer, represents the cross-correlation operation;
[0098] S23. According to the trained weights, the feature maps in the two branches are aggregated separately, and the output is sent to the classification-regression sub-network.
[0099] In specific implementation, as a preferred embodiment of the present invention, in step S22, the cross-layer correlation feature fusion module adopted is composed of three sub-modules with the same network architecture, which are used to calculate two double-layer feature correlation maps and one single-layer feature correlation map; each sub-module first projects the features to the classification and regression branches through parallel convolutional layers, and then uses deep cross-correlation to generate a response map representing the correlation between the inputs.
[0100] In specific implementation, as a preferred embodiment of the present invention, step S3 specifically includes:
[0101] S31, the classification branch consists of five 3×3 convolutional layers, generating a |P cls | w×h×1 The single-channel classification score map represented by is used to calculate the position correspondence between the search image S and the prediction map P as follows:
[0102]
[0103] Among them, (S x ,S y ) and (P x ,P y ) represent the coordinates of the sampled pixels in the search image and the prediction image, respectively, (s w ,s h ) and (w,h) represent the width and height of the search image and prediction image respectively, and s represents the step size;
[0104] S32, the regression branch directly outputs the distance from the pixel to the four sides of the target bounding box, using |P reg | w×h×1 Represents the regression map of the four channels and calculates the bounding box. The formula is as follows:
[0105]
[0106] Among them, O = (l, t, r, b) represents the offset of the pixel to the left, top, right and bottom boundaries of the target, and the coordinates of the upper left corner and lower right corner represent the rectangular bounding box.
[0107] In specific implementation, as a preferred embodiment of the present invention, step S4 specifically includes:
[0108] S41. Introduce the classification ranking loss function and calculate the classification loss. The process is as follows:
[0109] S411. Train a classifier supervised by cross entropy loss.
[0110] S412, sort all negative samples according to the predicted confidence scores, and sort the negative samples with confidence scores lower than τ reg The negative samples are filtered out, and the remaining negative samples constitute the difficult negative sample set where n - represents the number of negative samples, p j- Represents negative sample j - Confidence score of
[0111] S413. For positive samples, retain all positive samples to obtain a positive sample set
[0112] S414. Calculate the expectation of difficult negative samples and positive samples. The calculation formula is as follows:
[0113]
[0114] in, and Represent the weights of negative samples and positive samples respectively;
[0115] S415. Use the cross entropy loss function to calculate the classification loss as follows:
[0116]
[0117] Among them, α and β represent hyperparameters;
[0118] In this embodiment, there are two main reasons for not performing point-by-point comparison between the two sets: first, point-by-point comparison is very costly for training; second, it is not necessary to require that each positive sample be ranked higher than all negative samples; as long as one positive sample is ranked higher than the difficult negative sample, the tracker can select the appropriate sample as the tracking target. Therefore, sorting the expectations of the training samples can significantly reduce the time complexity.
[0119] S42, introduce the IoU-guided ranking loss function and calculate the classification loss. The process is as follows:
[0120] S421, for positive sample i,j∈A pos , set the sorting constraints as follows:
[0121]
[0122] Among them, p i and p j Represent the confidence scores of positive samples i and j respectively; Respectively represent the IoU values of positive samples i and j;
[0123] S422. Define the IoU-guided ranking loss function as:
[0124]
[0125] Among them, γ represents the hyperparameter that controls the loss.
[0126] In this embodiment, the IoU-guided ranking loss function can solve the problem of mismatch between classification confidence and positioning. The loss function can align the confidence scores of two positive samples with their corresponding IoUs, prompting a positive sample with a larger IoU to rank another positive sample with a smaller IoU.
[0127] In specific implementation, as a preferred embodiment of the present invention, step S5 specifically includes:
[0128] S51. The entire network is trained end-to-end on large-scale datasets, including COCO, DET, VID, YouTube-BB, GOT-10K and LaSOT. The template image size is 127×127 and the search image size is 255×255.
[0129] S52, initialize the backbone network using weights pre-trained on ImageNet;
[0130] S53, the network is trained using stochastic gradient descent with a batch size of 32 and 20 training rounds.
[0131] In specific implementation, as a preferred embodiment of the present invention, the single target tracking method based on cross-layer feature fusion and sorting loss is characterized in that step S6 specifically includes:
[0132] S61, in the testing phase, cutting out a template image from the first frame, inputting it into a feature extraction network, and caching the extracted template features;
[0133] S62, for subsequent frames, crop the image and extract features according to the target position of the previous frame, and then make predictions in the search area to obtain a classification score map and a regression map;
[0134] S63. After generating the prediction box, use the cosine window and scale change penalty to smooth the movement and change of the target, then select the prediction box with the highest score and update its size by linear interpolation with the state of the previous frame.
[0135] Example
[0136] OTB100 is a widely used public dataset consisting of 100 sequences. Figure 2 : is a graph of the success rate and accuracy of the tracker SiamXR proposed in the present invention and nine other trackers on the OTB100 dataset, wherein the tracker SiamXR proposed in the present invention has a success rate of 68.8% and an accuracy of 89.9%.
[0137] GOT-10K is a large-scale dataset for tracking general objects in the wild. It contains more than 10,000 video clips of real-world moving objects. The target categories in the training set and the test set are non-overlapping. The tracking results are automatically analyzed when uploaded to the official website. The evaluation indicators provided include average overlap rate (AO) and success rate (SR). AO represents the average overlap rate between the predicted bounding box and the real box, and SR represents the average overlap rate between the predicted bounding box and the real box. 0.5 Indicates the proportion of successfully tracked frames with an overlap ratio exceeding 0.5, SR 0.75 It indicates the proportion of successfully tracked frames with an overlap rate of more than 0.75. Table 1 shows the quantitative results of different indicators of the tracker SiamXR proposed in this invention and other trackers. Compared with SiamBAN, it improves 3.5% on AO and SR 0.5 The SR 0.75 The average annual growth rate was 3.3%.
[0138] Table 1 Quantitative results of different indicators of the tracker SiamXR proposed in this invention and other trackers
[0139]
[0140] LaSOT is a high-quality, large-scale dataset with a total of 1,400 tracking sequences. Compared with other datasets, LaSOT has longer sequences, which is a huge challenge for tracking algorithms. Figure 3 This is a graph of the success rate and accuracy of the tracker SiamXR proposed in this invention and nine other trackers on the LaSOT dataset. Compared with SiamBAN, the success rate is improved by 3.3% and the accuracy is improved by 3.8%.
[0141] UAV123 is an aerial video dataset containing 123 sequences captured from a low-altitude perspective. This dataset can be used to evaluate whether the tracker is suitable for deployment on UAVs in real-time scenarios. Figure 4 This is a graph of the success rate and accuracy of the tracker SiamXR proposed in this invention and nine other trackers on the UAV123 dataset. Compared with SiamBAN, the success rate is improved by 3.2% and the accuracy is improved by 4.9%.
[0142] In order to verify the role of each module in the embodiment of the present invention, an ablation experiment was conducted on the GOT-10K dataset. Figure 5The ablation experiment result diagram provided for the embodiment of the present invention. The scheme of the present invention mainly introduces the cross-layer feature fusion module and the ranking loss function module. SiamXR-NO represents that neither the cross-layer feature fusion module nor the ranking loss module is used; SiamXR-X represents that only the cross-layer feature fusion module is used; SiamXR-R represents that only the ranking loss function module is used; SiamXR represents that both the cross-layer feature fusion module and the ranking loss function module are used. The success rate of SiamXR-NO is 55.9%, the success rate of SiamXR-X using the cross-layer feature fusion module is improved by 2%, the success rate of SiamXR-R using the ranking loss function module is improved by 4.6%, and the success rate of SiamXR using both the cross-layer feature fusion module and the ranking loss function module is improved by 5.5%. The results show that each module has a certain degree of improvement in the tracking results, and their combination has a significant improvement in the model performance.
[0143] In order to explore the performance of the solution of the present invention in different tracking scenarios, experiments with 11 interference factors were conducted on the OTB-100 dataset. These interference factors include fast movement, out-of-plane rotation, in-plane rotation, target deformation, target out of view, occlusion, illumination change, background clutter, motion blur, scale change and low pixel count. Figure 6-Figure 16 The success rate and accuracy of the solution of the present invention under different interferences are shown in Figure 2. In most interference environments, the solution of the present invention has good performance and is capable of handling diverse challenges.
[0144] The performance of the scheme of the present invention in practical applications is observed through qualitative analysis. The Board, CarDark, Coke, Diving, Dog, Ironman, Liquor and Rubik sequences in the OTB-100 dataset are selected for analysis. These video sequences cover visual attributes such as scale change, target deformation, occlusion, out of field of view and illumination change. Fig.17 The figure shows the qualitative analysis results of the solution of the present invention and other tracking algorithms. In the Board video sequence with scale change interference, CFNet and SiamFC lost the target at the 48th frame; SiamBAN also lost the target at the 602nd frame. In the Liquor video sequence with background clutter interference, CFNet lost the target at the 377th frame, SiamBAN, SiamCAR and SiamFC failed to locate the real target, and at the 1575th frame, only SiamXR of the solution of the present invention can accurately locate the target position. In the Ironman video sequence, the lighting changes dramatically around the 145th frame, and the target is moving rapidly, but SiamXR can still maintain stable tracking of the target.
[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A single target tracking method based on cross-layer feature fusion and ranking loss, characterized in that: include: S1. Obtain a template image and a search image, and use a twin network to extract features from the template image and the search image to obtain template features and search features; S2, using the cross-layer feature fusion module, calculate the correlation between the template features and the search features to obtain the response map; S3. Send the obtained response map to the classification-regression subnetwork, where the classification branch is used to predict the category of each position and the regression branch is used to calculate the target bounding box of the position; S4, introduce the sorting loss function and calculate the loss; S5. Use the public datasets COCO, DET, VID, YouTube-BB, GOT-10K training set and LaSOT training set to train the model; S6. Test the model performance on the public datasets OTB-100, UAV123, LaSOT test set, and GOT-10K test set to predict the bounding box of the target.
2. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S1 specifically includes: S11. The improved ResNet-50 is used as the backbone network for feature extraction. Only the first four convolution blocks of ResNet-50 are retained. In order to retain detailed spatial information and increase the receptive field, all 3×3 convolution kernels in the fourth convolution block are expanded at a rate of 2 times, and the convolution step size is reduced from 2 to 1. S12 resizes the template image and search image to a specific size, encodes them into feature maps through ResNet-50 with the same parameters, and then reduces the number of feature channels to 256 through the neck network; S13, extract the template feature z and search feature x from the specific convolution layer, and define the template feature z as f z convi , define the search feature x as Where i=1,…,4.
3. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S2 specifically includes: S21. Define single-layer feature correlation as the correlation between two features at the same level, and define double-layer feature correlation as the correlation between features at two different levels. S22, using the cross-layer feature fusion module to calculate f z conv3 and The correlation between z conv4 and The correlation between z conv4 and The specific process is described as follows: in, and They represent the feature response maps of the regression branch and the classification branch respectively, (i,j)∈[(3,4),(4,3),(4,4)], Φ represents a 3×3 convolutional layer, represents the cross-correlation operation; S23. According to the trained weights, the feature maps in the two branches are aggregated separately, and the output is sent to the classification-regression sub-network.
4. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 2, characterized in that: In step S22, the cross-layer correlation feature fusion module used is composed of three sub-modules with the same network architecture, which is used to calculate two double-layer feature correlation maps and one single-layer feature correlation map; each sub-module first projects the features to the classification and regression branches through parallel convolutional layers, and then uses deep cross-correlation to generate a response map representing the correlation between the inputs.
5. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S3 specifically includes: S31, the classification branch consists of five 3×3 convolutional layers, generating a |P cls | w×h×1 The single-channel classification score map represented by is used to calculate the position correspondence between the search image S and the prediction map P as follows: Among them, (S x ,S y ) and (P x ,P y ) represent the coordinates of the sampled pixels in the search image and the prediction image, respectively, (s w ,s h ) and (w,h) represent the width and height of the search image and prediction image respectively, and s represents the step size; S32, the regression branch directly outputs the distance from the pixel to the four sides of the target bounding box, using |P reg | w×h×1 Represents the regression map of the four channels and calculates the bounding box. The formula is as follows: Among them, O = (l, t, r, b) represents the offset of the pixel to the left, top, right and bottom boundaries of the target, and the coordinates of the upper left corner and lower right corner represent the rectangular bounding box.
6. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S4 specifically includes: S41. Introduce the classification ranking loss function and calculate the classification loss. The process is as follows: S411. Train a classifier supervised by cross entropy loss. S412, sort all negative samples according to the predicted confidence scores, and sort the negative samples with confidence scores lower than τ reg The negative samples are filtered out, and the remaining negative samples constitute the difficult negative sample set Where n - represents the number of negative samples, p j- Represents negative sample j - Confidence score of S413. For positive samples, retain all positive samples to obtain a positive sample set S414. Calculate the expectation of difficult negative samples and positive samples. The calculation formula is as follows: in, and Represent the weights of negative samples and positive samples respectively; S415. Use the cross entropy loss function to calculate the classification loss as follows: Among them, α and β represent hyperparameters; S42, introduce the IoU-guided ranking loss function and calculate the classification loss. The process is as follows: S421, for positive sample i,j∈A pos , set the sorting constraints as follows: Among them, p i and p j Represent the confidence scores of positive samples i and j respectively; Respectively represent the IoU values of positive samples i and j; S422. Define the IoU-guided ranking loss function as: Among them, γ represents the hyperparameter that controls the loss.
7. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S5 specifically includes: S51. The entire network is trained end-to-end on large-scale datasets, including COCO, DET, VID, YouTube-BB, GOT-10K and LaSOT. The template image size is 127×127 and the search image size is 255×255. S52, initialize the backbone network using weights pre-trained on ImageNet; S53, the network is trained using stochastic gradient descent with a batch size of 32 and 20 training rounds.
8. A single target tracking method based on cross-layer feature fusion and sorting loss according to claim 1, characterized in that: Step S6 specifically includes: S61, in the testing phase, cutting out a template image from the first frame, inputting it into a feature extraction network, and caching the extracted template features; S62, for subsequent frames, crop the image and extract features according to the target position of the previous frame, and then make predictions in the search area to obtain a classification score map and a regression map; S63. After generating the prediction box, use the cosine window and scale change penalty to smooth the movement and change of the target, then select the prediction box with the highest score and update its size by linear interpolation with the state of the previous frame.