An attention-based single-target tracking method

Abstract

The application relates to an attention-based single-target tracking method, and belongs to the field of computer vision in deep learning. The method comprises data preprocessing, model training, training of a constructed network model, optimization of network parameters by reducing a loss function of the network during the whole training process, and obtaining network weights of an attention-based single-target network architecture; in a new video sequence, the network weights obtained through training are used to test the tracking target effect. The method has the advantages that low-level features and high-level features output by a backbone network are combined, the last feature map has high-level semantic information and supplements some detailed information, a classification enhancement module is used to assist a classification branch, the classification branch can obtain more accurate target positioning information, the increased design can enable the tracker to accurately track the target when similar objects interfere, and the tracker can accurately and stably track the target in numerous complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision in deep learning, and particularly relates to an attention-based single-object tracking method. Background Technology

[0002] Target tracking has always been a popular research area in computer vision, with wide applications in human-computer interaction, intelligent control, automatic navigation, and security monitoring. Although tracking methods have made some progress, their performance is still affected by complex scenarios such as changing lighting, non-rigid deformation, and interference from similar objects. Therefore, achieving long-term stable and accurate target tracking remains a challenging task.

[0003] Most mainstream object trackers are currently based on Siamese network architectures. This architecture transforms the object tracking problem into a matching problem, learning a general similarity mapping through cross-correlation operations between the target template and the search region. SINT was the first to apply Siamese networks to object tracking, transforming the object tracking problem into an image patch matching problem. In the same year, Bertinetto et al. proposed SiamFC, a novel fully convolutional Siamese network as the basic tracking algorithm, which maintains a certain level of accuracy while maintaining high speed. SiamRPN borrows from the Region Recommendation Network (RPN), feeding the feature map into classification and regression branches after feature extraction in the Siamese network, enabling the tracker to regress the target's position and shape. Subsequent methods such as DaSiamRPN, C-RPN, and SiamRPN++ have all made different improvements to SiamRPN. Almost all current state-of-the-art trackers use cross-correlation for information embedding. Although these trackers have achieved good tracking performance, they still experience object tracking errors when faced with interference from similar objects or changes in lighting conditions. The analysis of existing trackers yielded the following two conclusions: (1) Most Siam network-based trackers (such as SiamFC, SiamRPN, and ATOM) achieve similarity learning by performing convolutional cross-correlation operations between the target and the search region. However, such a method is itself a simple linear matching process, which leads to the loss of semantic information. (2) The cross-correlation features generated by the similarity mapping of Siam networks are easily interfered with by similar objects within the search region. Summary of the Invention

[0004] This invention provides an attention-based single-target tracking method. In deep learning-based methods, given the aforementioned shortcomings of some existing advanced Siamese network single-target trackers, this invention highlights the target information that needs attention by using attention. Furthermore, a classification enhancement module is added to the regression part to assist the classification part in classifying more accurately, thereby prompting the subsequent classification and regression sub-networks to more accurately locate the target and perform bounding box regression.

[0005] The technical solution adopted by this invention includes the following steps:

[0006] Step 1: Data preprocessing to prepare data for subsequent network model training;

[0007] Step 2: Model training. The constructed network model is trained. During the training process, the network parameters are optimized by reducing the network's loss function, thereby obtaining the network weights of the attention-based single-objective network architecture.

[0008] Step 3: Model testing. In new video sequences, the network weights obtained through training are used to test the tracking effect on the target.

[0009] The data preprocessing in step one of this invention includes:

[0010] S11. After preprocessing the data, each video image in each dataset is cropped into a pair of fixed sizes and placed in a newly generated folder. The folder contains cropped template images for training and sample images of the search region. The template image Z is 127×127 in size, and the search region image X is 511×511 in size.

[0011] The training of the model in step two of this invention includes the following steps:

[0012] S21. The prepared sample images are fed into the network model for training. This network model consists of three parts: a twin sub-network for feature extraction, a feature fusion structure that embeds information from the template image and the search area image, and a classification and regression sub-network for classification and regression of the target location.

[0013] S22. The first part is a twin network for feature extraction, which extracts features from the template image Z and the search region image X, respectively. The backbone network consists of two branches sharing the same CNN structure, with one branch taking the target template Z as input and the other branch taking the search region X as input. Two feature maps are obtained through the backbone network, namely the template image feature map. and search region feature map

[0014] S23. The second part is the feature fusion structure, used to embed features from the target template and the search region; it combines the feature maps of the two branches output by the backbone network. as well as Deep cross-correlation is performed to embed information from both layers. The response map R obtained through deep cross-correlation is then processed through an attention mechanism to extract the target information that needs to be focused on, thereby directing more attention to the target information region. Finally, the feature maps from layers 3, 4, and 5 are combined. Response map R to be stitched together all For subsequent classification and regression subnetworks, the deep cross-correlation and concatenation formulas are defined as follows:

[0015]

[0016] The above formula represents the deep cross-correlation operation, where and These represent the template feature map and the search region feature map, respectively.

[0017]

[0018] in, These represent the output feature maps of layers 3, 4, and 5 of the backbone network, respectively, and are the output parts obtained through deep cross-correlation and attention mechanisms.

[0019] S24. The third part is the classification and regression subnetwork, mainly used for feature map location classification and target location regression; first, the response map R... all Each position (i,j) in the response graph R can be mapped back to the search region (x,y). all Convolution will produce classification and regression branches; the classification branch will produce classification feature maps. and central feature map Classification feature maps are used to predict the category at each location. Each point (i,j,:) contains a 2D vector representing the corresponding foreground and background scores, respectively. An enhancement module is added to the classification part to assist in more accurate classification. Simultaneously, a center feature map runs parallel to this classification feature map. The algorithm gives a score for the center of each pixel. The higher the score, the center is the location. The center can be used to remove outliers. Locations far from the center often produce low-quality predicted bounding boxes.

[0020] S25. The regression branch of the classification regression network outputs the regression feature map. Regression Feature Map Each point (i,j,:) contains a 4D vector t(i,j) = (l,t,r,b), which represents the distance from the corresponding position to the four sides of the bounding box in the input search region. Let (x0,y0) and (x1,y1) represent the top-left and bottom-right corners of the ground truth bounding box, and (x,y) represent the position of point (i,j) on a certain point in the regression feature map. regression target It can be calculated using the following formula:

[0021]

[0022]

[0023] Where (x0, y0) and (x1, y1) represent the top left and bottom right corners of the truth bounding box. Represents the corresponding points on the regression feature map The goal of return, These represent the distances from a point on the regression feature map to the four sides of the bounding box, respectively.

[0024] S26. The entire network is trained using an end-to-end approach; the loss function value for the classification part is... The bounding box regression function value is The centrality loss is The weights are combined according to their respective weight values, and the weighted loss function is used for the entire system.

[0025]

[0026] In the above formula, cross-entropy loss is used for classification, IOU loss for regression, and centrality loss is also used.

[0027] S27. According to the loss function The gradient is calculated, and then the SGD optimizer is used to update the network parameters, so that the loss function of the entire network decreases until convergence. Then the entire training is completed, and a trained single-object tracking network weight based on attention is obtained.

[0028] In step S22 of this invention, the CNN network used is an improved ResNet-50. Specifically, the downsampling operation in the two convolutional blocks is removed, and dilated convolution is used to increase the receptive field. The effective stride of the last two blocks of conv4 and conv5 is modified to 1; the dilation rate of conv4 is set to 2, and the dilation rate of conv5 is set to 4; and an additional 1×1 convolutional layer is added after each block to reduce the output channels to 256.

[0029] In step S23 of this invention, the attention mechanism used is self-attention, the formula of which is as follows:

[0030]

[0031] Where Q represents the query value, K represents the key value, V represents the extracted information value, and d k The length of the vector is represented by the value. The subsequent Q-turn is matched with each K value, which is equivalent to calculating the correlation between features. Then, the softmax function is used for normalization to obtain a coefficient matrix. The greater the correlation, the greater the weight of the corresponding V information value.

[0032] In step S23 of this invention, the response maps of the features from the last 3, 4, and 5 layers of the backbone network after passing through the attention mechanism are stitched together. The purpose is to give the features richer semantic information and more detailed information, which helps to accurately locate the target.

[0033] In step S24 of this invention, the enhancement module first adjusts the channels through convolution, then embeds information through deep cross-correlation, and finally obtains the classification confidence score through convolution to help the classification part classify more accurately.

[0034] In step S26 of the present invention, the values ​​of λ1 and λ2 are 1 and 3, respectively.

[0035] The testing of the model in step three of this invention includes:

[0036] S31. Test the tracking effect of the trained weight parameters in a new video sequence.

[0037] In the target domain, reduced feature map resolution means a loss of detailed appearance features of the target, which is detrimental to similarity matching and easily leads to tracking errors. This invention utilizes attention to highlight target information, and the method uses contextual information to enhance the original features; furthermore, it adds detailed information to the output of the last three layers of feature maps used in the backbone network, making the extracted features richer.

[0038] The advantages of this invention are that it combines the low-level features output by the backbone network with the high-level features, so that the final feature map has high-level semantic information while supplementing some detailed information; furthermore, it uses a classification enhancement module to assist the classification branch, so as to obtain more accurate target localization information. The added design allows the tracker to perform accurate tracking even when there is interference from similar objects. In the test comparison experiment, this invention has a more robust tracking effect compared with other methods, and can accurately and stably track targets in many complex scenarios. Attached Figure Description

[0039] Figure 1 This is a network structure block diagram of the present invention;

[0040] Figure 2 This is a flowchart illustrating the structure of the enhanced module of the present invention;

[0041] Figure 3 It is a pair of template and search area sample images;

[0042] Figure 4 This is a comparison chart of the results of different algorithms on the GOT-10k test dataset. Detailed Implementation

[0043] To further understand the various parts of this invention, the following description and explanation are provided in conjunction with the accompanying drawings to make the invention clearer.

[0044] like Figure 1 As shown, it includes the following steps:

[0045] Step 1: Data preprocessing to prepare data for subsequent network model training;

[0046] Step 2: Model training. The constructed network model is trained. During the training process, the network parameters are optimized by reducing the network's loss function, thereby obtaining the network weights of the attention-based single-objective network architecture.

[0047] Step 3: Model testing. In new video sequences, the network weights obtained through training are used to test the tracking effect on the target.

[0048] The specific data preprocessing operations mentioned above include the following:

[0049] S11. After data preprocessing, each video image in each dataset is cropped into a pair of fixed-size images and placed in a newly generated folder. This folder contains cropped template images for training and sample images for the search region. The template image Z is 127×127 pixels, and the search region image X is 511×511 pixels. See [link to relevant documentation]. Figure 3 ;

[0050] The training of the model includes the following steps:

[0051] S21. The prepared sample images are fed into the network model for training. This network model consists of three parts: a twin sub-network for feature extraction, a feature fusion structure that embeds information from the template image and the search area image, and a classification and regression sub-network for classification and regression of the target location.

[0052] S22. The first part is a twin network for feature extraction, which extracts features from the template image Z and the search region image X, respectively. The backbone network consists of two branches sharing the same CNN structure, with one branch taking the target template Z as input and the other branch taking the search region X as input. Two feature maps are obtained through the backbone network, namely the template image feature map. and search region feature map

[0053] S23. The second part is the feature fusion structure, used to embed features from the target template and the search region; it combines the feature maps of the two branches output by the backbone network. as well as Deep cross-correlation is performed to embed information from both layers. The response map R obtained through deep cross-correlation is then processed through an attention mechanism to extract the target information that needs to be focused on, thereby directing more attention to the target information region. Finally, the feature maps from layers 3, 4, and 5 are combined. Response map R to be stitched together all For subsequent classification and regression subnetworks, the deep cross-correlation and concatenation formulas are defined as follows:

[0054]

[0055] The above formula represents the deep cross-correlation operation, where and These represent the template feature map and the search region feature map, respectively.

[0056]

[0057] in, These represent the output feature maps of layers 3, 4, and 5 of the backbone network, respectively, and are the output parts obtained through deep cross-correlation and attention mechanisms.

[0058] S24. The third part is the classification and regression subnetwork, mainly used for feature map location classification and target location regression; first, the response map R... all Each position (i,j) in the response graph R can be mapped back to the search region (x,y). all Convolution will produce classification and regression branches; the classification branch will produce classification feature maps. and central feature map Classification feature maps are used to predict the category at each location. Each point (i,j,:) contains a 2D vector representing the corresponding foreground and background scores, respectively. An enhancement module is added to the classification part to assist in more accurate classification. Simultaneously, a center feature map runs parallel to this classification feature map. The algorithm gives a score for the center of each pixel. The higher the score, the center is the location. The center can be used to remove outliers. Locations far from the center often produce low-quality predicted bounding boxes.

[0059] S25. The regression branch of the classification regression network outputs the regression feature map. Regression Feature Map Each point (i,j,:) contains a 4D vector t(i,j) = (l,t,r,b), which represents the distance from the corresponding position to the four sides of the bounding box in the input search region. Let (x0,y0) and (x1,y1) represent the top-left and bottom-right corners of the ground truth bounding box, and (x,y) represent the position corresponding to point (i,j). A point on the regression feature map... regression target It can be calculated using the following formula:

[0060]

[0061]

[0062] Where (x0, y0) and (x1, y1) represent the top left and bottom right corners of the truth bounding box. Represents the corresponding points on the regression feature map The goal of return, These represent the distances from a point on the regression feature map to the four sides of the bounding box, respectively.

[0063] S26. The entire network is trained using an end-to-end approach; the loss function value for the classification part is... The bounding box regression function value is The centrality loss is The weights are combined according to their respective weight values, and the weighted loss function is used for the entire system.

[0064]

[0065] In the above formula, cross-entropy loss is used for classification, IOU loss for regression, and centrality loss is also used.

[0066] S27. According to the loss function The gradient is calculated, and then the SGD optimizer is used to update the network parameters, so that the loss function of the entire network decreases until convergence. Then the entire training is completed, and a trained single-object tracking network weight based on attention is obtained.

[0067] In step S22, the backbone network CNN uses a modified ResNet-50, which removes the downsampling operation in the last two convolutional blocks and uses dilated convolutions to increase the receptive field. Specifically, the effective stride of the last two blocks of conv4 and conv5 is modified to 1; the dilation rate of the conv4 block is set to 2 and the dilation rate of the conv5 block is set to 4; and an additional 1×1 convolutional layer is added after each block to reduce the output channels to 256, as shown in Table 1.

[0068] Table 1 shows the specific settings for the parameters of the last three layers of the backbone network.

[0069]

[0070] In step S23, the attention mechanism used is self-attention, the formula of which is as follows:

[0071]

[0072] In the formula, Q represents the query value, K represents the key value, V represents the extracted information value, and d k The length of the vector is represented by the Q-turn and each K value is matched, which is equivalent to calculating the correlation between features. Then, the softmax function is used to normalize it, resulting in a coefficient matrix. The greater the correlation, the greater the weight of the corresponding V information value.

[0073] In step S23, the response maps of the features from the last 3, 4, and 5 layers of the backbone network after passing through the attention mechanism are stitched together. The purpose is to give the features richer semantic information and more detailed information, which helps to accurately locate the target.

[0074] The enhancement module in step S24 is as follows: Figure 2 As shown, the process involves first adjusting the channels through convolution, then embedding information through deep cross-correlation, and finally obtaining the classification confidence score through convolution to help the classification part to classify more accurately.

[0075] In step S26, λ1 and λ2 take values ​​of 1 and 3, respectively.

[0076] The testing of the model in step three includes the following steps:

[0077] S31. Test the tracking effect of the trained weight parameters in a new video sequence that has never appeared before.

[0078] In target domains, reduced feature map resolution means a loss of detailed appearance features of the target, which is detrimental to similarity matching and easily leads to tracking errors. This invention combines low-level features from the backbone network output with high-level features, allowing the final feature map to possess high-level semantic information while supplementing it with detailed information. Furthermore, a classification enhancement module assists the classification branch, resulting in more accurate target localization information. This added design allows the tracker to perform accurate tracking even when interfered with by similar objects. In comparative testing, this invention demonstrates more robust tracking performance compared to other methods and can accurately and stably track targets in numerous complex scenarios.

[0079] The network model was trained using data provided on the GOT-10k official website, and the training effect of the proposed method was tested using the evaluation tools for the GOT-10k dataset. The data in Table 2 shows that the single-target tracking algorithm proposed in this invention exhibits superior performance on the test data compared to other algorithms trained with the same weight parameters.

[0080] Table 2 Evaluation of the GOT-10k test dataset

[0081]

[0082] To provide a more direct visual experience, the test results are displayed on a video sequence and compared with some previous algorithms, as follows: Figure 4 As shown, bounding boxes with different arc grayscale levels represent the test results of different tracking algorithms on the GOT-10k dataset. In the first video sequence, it can be seen that the SiamRPN++ algorithm made a tracking error, mistaking the shape on the shore for a boat and a yacht and including it in the bounding box. In the second video sequence, it can be seen that the animal's color is extremely similar to the surrounding environment, and both SiamRPN++ and ECO made tracking errors, while SiamCAR only tracked the animal's head, while the tracking algorithm of this invention tracked the entire animal. In the third video sequence, ECO completely lost track, SiamRPN++ tracked part of the object, and the algorithm proposed in this invention was more successful. Therefore, the algorithm proposed in this invention can achieve long-term stable tracking results in different complex scenes.

[0083] The above embodiments are the preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the scope of protection of this aspect. More specifically, within the scope of the disclosure, drawings, and claims of this application, various modifications and improvements can be made to the components and / or layout of the subject matter combination layout.

Claims

1. An attention-based single object tracking method, characterized in that, Comprise the following steps: Step one: data preprocessing, provide data preparation for subsequent network model training; Step two: model training, the network model is trained, and the network parameters are optimized by reducing the loss function of the network during the whole training process, so as to obtain the network weight of the single target network architecture based on attention; The model training comprises the following steps: S21, the prepared sample picture is sent into the network model for training, the network model comprises three parts: one is a twin subnetwork for feature extraction, one is a feature fusion structure for embedding template picture and search area picture information, and one is a classification and regression subnetwork for classification and regression target position; S22, the first part is a twin sub-network for feature extraction, which extracts features from the template picture Z and the search region picture X respectively; the backbone network is shared by two branches with the same CNN structure, wherein one branch takes the template Z as input and the other branch takes the search region X as input; two feature maps are obtained through the backbone network, which are the template image feature map and the search region feature map ; S23, the second part is a feature fusion structure, used for embedding features of the target template and the search region; the two branch feature maps output by the backbone network are fused and deep cross-correlation is performed to embed the information of both, and a response map obtained through deep cross-correlation is obtained a target information needing attention is extracted through an attention mechanism, so as to pay more attention to the target information region, and finally the 3, 4 and 5 layer feature maps are spliced , i=3, 4, 5, the spliced response map used for subsequent classification regression sub-networks, deep cross-correlation and splicing sub-networks are defined as follows: ； The above equation represents a deep cross-correlation operation, where with denote the template feature map and the search region feature map, respectively; ； wherein, , , respectively represent the backbone network 3, 4, 5 layer output feature maps, output part through deep cross-correlation and attention mechanism; S24. The third part is the classification and regression subnetwork, mainly used for feature map location classification and target location regression; first, the response map... Each position in Map back to the search region Response diagram Convolution will produce classification and regression branches; The classification branch will get the classification feature map and the center feature map The classification feature map is used to predict the category of each position, and the classification feature map Each point contains a 2D vector, which respectively represents the corresponding foreground and background score, and an enhancement module is added in the classification part to assist the classification to be more accurate; At the same time, there is also a center feature map parallel to the classification feature map, and the center feature map gives the score of the center of each pixel point, and the high score is the center position. The center score is used to delete outliers, and the position far away from the center will often produce a low-quality prediction bounding box; The enhanced module is specifically: a feature map and a feature map First, the channels are adjusted through convolution operation, then information is embedded through deep cross-correlation operation, and finally, classification confidence is obtained through convolution operation to assist classification part, so that the classification result is more accurate. S25, the regression branch of the classification regression network outputs a regression feature map , the regression feature map Each point contains a 4D vector , which represents the distance from the corresponding position to the four sides of the bounding box in the input search area, set and represent the upper left corner and the lower right corner of the true value bounding box, represent the point corresponding position, the regression target of a certain point on the regression feature map is calculated by the following formula: ； wherein and denote the top-left and bottom-right corners of the ground truth bounding box, denote the regression targets of the corresponding points on the regression feature map, denote the distances from the points on the regression feature map to the four sides of the bounding box, respectively. S26, training the whole network in an end-to-end manner; wherein the loss function value of the classification part is , the bounding box regression function value is , the centrality loss is , and the above are weighted together according to corresponding weight values as the weighted loss function of the whole system; ； In the formula, cross entropy loss is used for classification, IOU loss is used for regression, and center loss is also used; S27、According to the loss function the gradient is calculated, and the optimizer SGD is used to update the parameters of the network, so that the loss function of the entire network is reduced until convergence, and then the entire training is completed, and a trained network weight of the attention-based single-target tracking is obtained. Step three: model test, using the network weight obtained by training to test the tracking target effect in the new video sequence.

2. The attention-based single object tracking method of claim 1, wherein, The data preprocessing in the step one comprises: S11, each video picture in each data set is cropped to a pair of fixed sizes through a data preprocessing operation and placed in a regenerated folder, the folder contains only the sample pictures of the cropped templates and search areas used for training, wherein the size of the template picture Z is , and the size of the search area picture X is .

3. The attention-based single object tracking method of claim 1, wherein: The CNN network in the step S22 adopts an improved Resnet-50, specifically, the down-sampling operation in two convolution blocks is removed, and a hollow convolution is used to increase the receptive field, and the receptive field is increased by modifying and The effective stride of the last two blocks is set to 1; the The hollow rate of the block is set to 2, The hollow rate of the block is set to 4; an additional convolution layer is attached after each block, so that the output channel is reduced to 256.

4. The attention-based single object tracking method of claim 1, wherein: In the step S23, self-attention is used in the attention mechanism, and the formula is as follows: ； Where Q represents the query value, K represents the key value, and V represents the extracted information value, The length of the vector, and the subsequent Q returns and each K value are matched, which is to calculate the correlation between the characteristics, and then normalized by the softmax function to obtain a coefficient matrix. The greater the correlation, the greater the weight of the V information value.

5. The attention-based single object tracking method of claim 1, wherein: In the step S23, the response map after the attention mechanism is used to splice the features of the 3rd, 4th and 5th layers of the backbone network, so that the features have more semantic information and more detailed information, which is helpful for accurate positioning of the target.

6. The attention-based single object tracking method of claim 1, wherein: The step S26 With the values of 1 and 3, respectively.

7. The attention-based single object tracking method of claim 1, wherein: The model test in the step three comprises: S31, the trained weight parameters are used to test the tracking effect in a new video sequence.

Images