Visual object tracking method based on feature remapping

Abstract

The present invention is a visual target tracking method based on feature remapping, and the steps include: 1) the backbone network extracts template features and search area features respectively, calculates deep cross-correlation, and generates classification feature maps and regression feature maps through classification branches and regression branches; 2) constructs an interference assessment network to obtain a response map and analyzes whether there is similar target or clutter interference; 3) activates the feature remapping network to obtain remapped template features and remapped search area features, performs deep cross-correlation calculation, and generates a remapped classification response map; 4) fuses the remapped classification response maps, solves the interference and obtains the final tracking target position; 5) collects the interference set and target set online and constructs an online sample set, calculates the loss of the online sample set, and updates the parameters of the feature remapping network through gradient backpropagation. The method of the present invention improves the discriminability of target features through remapping and enhances the anti-interference performance of the tracking method.

Description

Technical Field

[0001] The present invention belongs to target tracking technology in the field of computer vision, and relates to a visual target tracking method based on feature remapping. Background Art

[0002] Visual tracking is one of the most fundamental tasks in computer vision, widely used in fields such as video surveillance, autonomous driving, and intelligent transportation. The primary goal is to identify an arbitrary target state (including its position, scale, and initial velocity) in the first frame of a video sequence, and then determine its position, velocity, and trajectory in each subsequent frame to achieve continuous and stable tracking.

[0003] In recent years, convolutional neural networks have achieved tremendous success in the field of vision. CNNs have strong feature representation and generalization capabilities, significantly improving tracker performance. However, due to the complexity of the real world, targets are often surrounded by numerous similar objects, occlusions, and significant background changes. The features extracted by the tracker contain a lot of interference information, which reduces the tracker's ability to discern target features, leading to misjudgments and tracking drift. As tracking progresses, drift errors accumulate, resulting in tracking loss / failure and affecting tracker performance.

[0004] Therefore, how to provide more discriminative features in interference scenarios and enhance the tracker's ability to handle interference areas for the current task is the key to improving visual target tracking performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a visual target tracking method based on feature remapping, which solves the problem that the tracking method in the prior art has insufficient feature discrimination ability when facing similar targets and clutter interference.

[0006] The technical solution adopted by the present invention is a visual target tracking method based on feature remapping, which is specifically implemented according to the following steps:

[0007] Step 1: SiamCAR, which has both speed and accuracy, is used as the base tracker. The base tracker consists of three parts: the backbone network φ, the classification branch, and the regression branch. The backbone network φ extracts the template feature φ(z) and the search area feature φ(x) respectively, calculates the "deep cross-correlation" of the template feature φ(z) and the search area feature φ(x), and then generates the classification feature map and regression feature map respectively after preprocessing by the classification branch and the regression branch.

[0008] Step 2: Construct an interference assessment network and transform the classification feature map H of each frame into cls (z,x) is sent to the interference assessment network, and 1*1 convolution is used to process multi-channel feature information to obtain the response map D cls(z,x), according to the response graph D cls The peak relationship analysis in (z, x) shows whether there are similar targets or clutter interference;

[0009] Step 3: When a large amount of interference occurs around the target, the feature remapping network ψ is enabled to process the template feature φ(z) and the search area feature φ(x), and the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are obtained. Then, the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are subjected to a “deep cross-correlation” classification embedding calculation to generate the remapping classification response map EH. cls (z,x);

[0010] Step 4: The classification response map of the base tracker SiamCAR is not good enough for interference scenes, so the fusion remap classification response map EH is used. cls (z,x), as the final classification response map, solve the final classification response map to obtain the final tracking target position;

[0011] Step 5: Collect interference information near the target online as the interference set The area where the target information is located is the target set Interference set and target set Construct an online sample set, calculate the loss of the online sample set, and backpropagate the gradient to update the parameters of the feature remapping network ψ.

[0012] The beneficial effects of the present invention include the following aspects:

[0013] 1) This paper analyzes the tracking performance characteristics when a large number of interference scenarios occur, and uses the similarity of the response peaks between the target and the interference to design an interference assessment network, so that the tracker can determine whether interference occurs, further optimize the tracking structure, and improve the tracking performance.

[0014] 2) The feature remapping network proposed in this paper obtains features with strong discriminative ability in the remapping space, which helps the tracker cope with situations where targets are similar and clutter interferes, and improves the tracker's anti-interference performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a flow chart of the visual target tracking method of the present invention;

[0016] Figure 2 This is a structural diagram of visual target tracking based on feature remapping in the method of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0018] Reference Figure 1 The visual target tracking method based on feature remapping of the present invention is specifically implemented according to the following steps:

[0019] Step 1: SiamCAR, which has both speed and accuracy, is used as the base tracker. The base tracker consists of three parts: the backbone network φ, the classification branch, and the regression branch. The backbone network φ of the base tracker extracts the template feature φ(z) and the search area feature φ(x) respectively, calculates the "deep cross-correlation" of the template feature φ(z) and the search area feature φ(x), and then generates the classification feature map and the regression feature map accordingly after preprocessing by the classification branch and the regression branch; the classification feature map is used to discriminate the foreground-background score, obtain the classification response map, and determine the target position; the regression feature map analyzes the bounding box mapping relationship of each position to obtain the regression response map, thereby predicting the target scale.

[0020] The specific process is:

[0021] 1.1) The backbone network φ of SiamCAR has two network branches with shared parameters, which respectively extract the features of the target template z in the first frame and the search area x in the subsequent frames. Here, the third and fifth layer features of the ResNet network are used, and the template features φ(z) and the search area features φ(x) are aggregated by cross-correlation operation to generate the aggregated feature map H(z,x), which is expressed as:

[0022] H(z,x)=φ(z)*φ(x) (1)

[0023] 1.2) Both the classification branch and the regression branch use a 4-layer reconstructed network CNN to preprocess the aggregate feature map H(z,x), and adjust the aggregate feature map structure to obtain the classification feature map H cls (z,x) and regression feature map H reg (z,x), and then perform subsequent classification and regression processing respectively, and output the classification response map of each pixel position and regression response plots The expression is:

[0024]

[0025] Among them, CNet i (i=cls,reg) represents the adjustment network CNN, on which the target position pos of the corresponding space of the backbone network φ is determined. φ .

[0026] Step 2: Construct an interference assessment network and transform the classification feature map H of each frame into cls (z,x) is sent to the interference assessment network, and 1*1 convolution is used to process multi-channel feature information to obtain the response map D of the target feature distribution. cls (z,x), according to the response graph Dcls The peak relationship analysis in (z, x) shows whether there are similar targets or clutter interference;

[0027] The specific process is:

[0028] 2.1) Construct an interference assessment network, which includes two parts: feature map channel fusion and interference assessment. The interference assessment network uses a 1*1 convolution module and a 3-layer CNN assessment module. The 1*1 convolution module is used to process the classification feature map H in step 1. cls (z,x), fuse multi-channel feature information to obtain the response map D about the target feature distribution cls (z,x);

[0029] 2.2) Search response graph D cls The maximum peak point P in (z,x) main =[R main ] max and the secondary peak point P sec =[R sec ] max , calculate the ratio of the two and the interference threshold T d Relationship, determine whether there is interference in the current frame, the expression is:

[0030]

[0031] Among them, the 3*3 area around the maximum peak point is selected to represent the main peak area set R main , select the 5*5 area around the secondary peak point to represent the secondary peak area set R sec ; In addition, 0 on the right side of formula (3) indicates no interference, and 1 indicates interference.

[0032] Step 3: When a large amount of interference occurs around the target, the feature remapping network ψ is enabled to process the template feature φ(z) and the search area feature φ(x), and the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are obtained. Then, the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are subjected to a “deep cross-correlation” classification embedding calculation to generate the remapping classification response map EH. cls (z,x), the specific process is:

[0033] 3.1) Construct a feature remapping network ψ to enhance target-related features.

[0034] The feature remapping network ψ uses the ResNet18 architecture. This step uses the features of Conv1-Conv4. Table 1 shows the ResNet-18 network structure and parameters.

[0035] Table 1. Parameters of the feature remapping network

[0036]

[0037]

[0038] 3.2) Input the template feature φ(z) and search area feature φ(x) of the corresponding space of the backbone network φ of the base tracker into the feature remapping network ψ, and obtain the enhanced template feature ψ[φ(z)] and enhanced search area feature ψ[φ(x)] respectively;

[0039] 3.3) Referring to formula (2), the calculation is performed in the feature remapping network ψ corresponding space, and the expression is:

[0040] EH cls (z,x)=ψ[φ(z)]*ψ[φ(x)] (4)

[0041] Where, EH cls (z,x) represents the remapped classification response map EH cls (z,x), according to the foreground-background classification probability, the target position pos in the corresponding space of the feature remapping network ψ is calculated. ψ .

[0042] Step 4: The classification response map of the base tracker SiamCAR is not good enough for interference scenes, so the fusion remap classification response map EH is used. cls (z,x), as the final classification response map, solve the final classification response map to obtain the final tracking target position. The specific process is:

[0043] 4.1) Linear weighted fusion of classification response maps and remapped categorical response map EH cls (z,x), the expression is:

[0044] RH cls (z,x)=λH cls (z,x)+(1-λ)EH cls (z,x) (5)

[0045] Among them, RH cls (z,x) represents the final classification response map, and the hyperparameter λ represents the weighted fusion factor, which controls the contribution of different classification responses;

[0046] 4.2) Analyze the foreground-background classification probability in the final classification response map to obtain the final tracking target position pos.

[0047] Step 5: Based on step 4, collect the interference information near the target online as the interference set The area where the target information is located is the target set Interference set and target set Construct an online sample set, calculate the loss of the online sample set, and backpropagate the gradient to update the parameters of the feature remapping network ψ to improve the adaptability of the feature remapping network to the interference scene of the current video sequence.

[0048] Experimental verification:

[0049] Reference Figure 2 , implement according to step 1 to step 4 of the method of the present invention, and pay attention to the following points:

[0050] In step 1, the data of three datasets GOT10K, COCO and TrackingNet are combined to train the base tracker SiamCAR, in which the model is initialized using the ImageNet pre-trained weights and trained for 20 rounds. In the first 10 rounds, Backbone is frozen to optimize the classification branch and regression branch, and in the last 10 rounds, Backbone, classification branch and regression branch are jointly optimized.

[0051] In step 2, the feature remapping network is trained offline on the GOT10K dataset. First, the SiamCAR-based tracker trained in step 1 is run to obtain tracking response results on different video sequences of GOT10K. Then, the peak value and interference threshold T d The target and interference regions were collected from the original GOT10K dataset to construct a training set. Finally, a gradient descent algorithm was used to learn the network parameters. The training images were uniformly resized to 127 x 127 pixels, and training was performed for 8,000 epochs. The learning rate was set to decay exponentially with the number of iterations.

[0052] In step 3, the base tracker SiamCAR and feature remapping network trained offline in steps 1 and 2 are loaded, and the target area of ​​the first frame of the sequence is cropped to a size of 107×107×3 as a template area, which is fed into the backbone network of the base tracker SiamCAR to extract the template features; according to the target position of the first frame, a size of 255×255×3 is cropped on the second frame of the sequence as a search area, which is fed into the backbone network of the base tracker SiamCAR to extract the search area features; through a series of processing of "deep cross-correlation" calculation, feature remapping, classification branch and regression branch, the target position of the second frame is determined, and this process is repeated to finally achieve stable tracking of the entire sequence.

[0053] Steps 1 to 4 of the proposed method were performed on the GOT10K test set to obtain tracking results. Average overlap (AO), a success rate (SR0.5) exceeding 0.5, and a success rate (SR0.75) exceeding 0.75 were used as evaluation metrics to evaluate the performance of the feature remapping-based visual object tracking method and the basis tracking SiamCAR. Table 2 shows a performance comparison on the GOT10K test set.

[0054] Table 2. GOT10K test set performance comparison table

[0055]

Claims

1. A visual target tracking method based on feature remapping, characterized in that: Follow these steps to implement: Step 1: SiamCAR, which has both speed and accuracy, is used as the base tracker. The base tracker consists of three parts: the backbone network φ, the classification branch, and the regression branch. The backbone network φ extracts the template feature φ(z) and the search area feature φ(x) respectively, calculates the "deep cross-correlation" of the template feature φ(z) and the search area feature φ(x), and then generates the classification feature map and regression feature map respectively after preprocessing by the classification branch and regression branch. Step 2: Construct an interference assessment network and transform the classification feature map H of each frame into cls (z,x) is sent to the interference assessment network, and 1*1 convolution is used to process multi-channel feature information to obtain the response map D cls (z,x), according to the response graph D cls The peak relationship analysis in (z, x) shows whether there are similar targets or clutter interference; Step 3: When a large amount of interference occurs around the target, the feature remapping network ψ is enabled to process the template feature φ(z) and the search area feature φ(x), and the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are obtained. Then, the remapping template feature ψ[φ(z)] and the remapping search area feature ψ[φ(x)] are subjected to a "deep cross-correlation" classification embedding calculation to generate the remapping classification response map EH. cls (z,x); Step 4: The classification response map of the base tracker SiamCAR is not good enough for interference scenes, so the fusion remap classification response map EH is used. cls (z,x), as the final classification response map, solve the final classification response map to obtain the final tracking target position; Step 5: Collect interference information near the target online as the interference set The area where the target information is located is the target set Interference set and target set Construct an online sample set, calculate the loss of the online sample set, and backpropagate the gradient to update the parameters of the feature remapping network ψ.

2. The visual target tracking method based on feature remapping according to claim 1, characterized in that: In step 1, the specific process is: 1.1) The backbone network φ of SiamCAR has two network branches with shared parameters, which respectively extract the features of the target template z in the first frame and the search area x in the subsequent frames. Using the third and fifth layer features of the ResNet network, the template features φ(z) and the search area features φ(x) are aggregated by cross-correlation operation to generate the aggregated feature map H(z,x), which is expressed as: H(z,x)=φ(z)*φ(x) (1) 1.2) Both the classification branch and the regression branch use a 4-layer reconstructed network CNN to preprocess the aggregate feature map H(z,x), and adjust the aggregate feature map structure to obtain the classification feature map H cls (z,x) and regression feature map H reg (z,x), and then perform subsequent classification and regression processing respectively, and output the classification response map of each pixel position and regression response plots The expression is: Among them, CNet i (i=cls,reg) represents the adjustment network CNN, on which the target position pos of the corresponding space of the backbone network φ is determined. φ .

3. The visual target tracking method based on feature remapping according to claim 1, characterized in that: In step 2, the specific process is: 2.1) Construct an interference assessment network, which includes two parts: feature map channel fusion and interference assessment. The interference assessment network uses a 1*1 convolution module and a 3-layer CNN assessment module. The 1*1 convolution module is used to process the classification feature map H in step 1. cls (z,x), fuse multi-channel feature information to obtain the response map D about the target feature distribution cls (z,x); 2.2) Search response graph D cls The maximum peak point P in (z,x) main =[R main ] max and the secondary peak point P sec =[R sec ] max , calculate the ratio of the two and the interference threshold T d Relationship, determine whether there is interference in the current frame, the expression is: Among them, the 3*3 area around the maximum peak point is selected to represent the main peak area set R main , select the 5*5 area around the secondary peak point to represent the secondary peak area set R sec ; In addition, 0 on the right side of formula (3) indicates no interference, and 1 indicates interference.

4. The visual target tracking method based on feature remapping according to claim 1, characterized in that: In step 3, the specific process is: 3.1) Construct a feature remapping network ψ to enhance target-related features. The feature remapping network ψ uses the ResNet18 architecture, and this step uses the features of Conv1-Conv4; 3.2) Input the template feature φ(z) and search area feature φ(x) of the corresponding space of the backbone network φ of the base tracker into the feature remapping network ψ, and obtain the enhanced template feature ψ[φ(z)] and enhanced search area feature ψ[φ(x)] respectively; 3.3) Referring to formula (2), the calculation is performed in the feature remapping network ψ corresponding space, and the expression is: EH cls (z,x)=ψ[φ(z)]*ψ[φ(x)] (4) Where, EH cls (z,x) represents the remapped classification response map EH cls (z,x), according to the foreground-background classification probability, the target position pos in the corresponding space of the feature remapping network ψ is calculated. ψ .

5. The visual target tracking method based on feature remapping according to claim 1, characterized in that: In step 4, the specific process is: 4.1) Linear weighted fusion classification response map and remapped categorical response map EH cls (z,x), the expression is: RH cls (z,x)=λH cls (z,x)+(1-λ)EH cls (z,x) (5) Among them, RH cls (z,x) represents the final classification response map, and the hyperparameter λ represents the weighted fusion factor, which controls the contribution of different classification responses; 4.2) Analyze the foreground-background classification probability in the final classification response map to obtain the final tracking target position pos.

Images