A target tracking method based on multi-layer attention feature enhancement

Abstract

The application relates to a target tracking method based on multi-layer attention feature enhancement, and belongs to the target tracking field, and comprises the following steps: S1, a target tracking network based on a twin network is constructed, a modified ResNet50 is used as a backbone network, and feature extraction is carried out on a template picture and a search picture; S2, a feature enhancement module is embedded in the last three layers in the backbone network to selectively enhance useful features; S3, the template features and the search features extracted by the backbone network are subjected to cross-correlation operation to obtain a response graph; S4, the cross-correlation feature graph is subjected to classification and regression, and the best tracking frame is obtained in multiple prediction frames, so that tracking is realized, and the template picture is updated online. The application makes feature interaction more frequent, enhances the expression capability of features, makes the tracker obtain more accurate target positioning information, and greatly resists the interference of similar backgrounds.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of target tracking, and relates to a target tracking method based on multi-layer attention feature enhancement. BACKGROUND

[0002] Target tracking is an important branch in the field of computer vision research, and is a prerequisite for realizing upper-layer tasks such as visual cognition and reasoning, and has a wide range of applications in fields such as video monitoring, unmanned driving, visual navigation and human-computer interaction. The core problem of target tracking is to distinguish the foreground target from the complex background, and only given the position of any target in the first frame, the tracker aims to successfully distinguish and locate the target in the subsequent frames. According to the difference of network model, the target tracking method can be divided into two categories: the first category is a target tracking method based on correlation filtering, and the second category is a target tracking method based on deep learning.

[0003] The target tracking method based on correlation filtering: first, sampling the target template, training the regressor based on these samples, then, using the regressor and the target candidate region to do correlation operation and generate response value, finally, finding the region corresponding to the maximum response value in the response map generated by the regressor and the candidate region, which can realize the tracking of the target. This method introduces cyclic sampling and kernel function, and its advantage lies in that the use of cyclic sampling method greatly increases the sampling efficiency and sample quality, and the introduction of kernel function enables the algorithm to efficiently map the original features to a complex high-dimensional space, greatly speeding up the solution process of the correlation filtering template. However, these samples generated by cyclic shift are periodic themselves, so they will cause marginal effect, which has two negative effects on tracking effect, first, it will reduce the discriminative ability of the filter template to the target in the training stage, second, in the detection and tracking stage, if a fast-moving target is encountered, the marginal effect will cause the target to have very small effect on the filter template, thus losing the target. The existing correlation filtering tracking method is a linear weighted update model with fixed learning rate, which does not need to explicitly save the training samples, the model trained by each frame of samples and the existing target model are weighted with fixed weights to update the target model, so the information of previous samples will gradually become invalid, and the information of the last few frames of samples accounts for a large proportion of the model. If the target positioning is inaccurate, occluded, disturbed by background, etc., the fixed learning rate method will treat these "problematic" samples equally, and the target model will be contaminated, resulting in tracking failure.

[0004] Deep learning based tracking method: with the rapid development of computer vision and big data technology, deep learning technology is widely used in various visual tasks, and high-performance target tracking algorithms based on deep learning are constantly proposed. Among them, the tracking algorithm using convolutional neural network to extract features and considering target tracking as a similarity measurement through the twin network framework has become the main direction of current research. Among them, SiamFC series models are widely used and many improved versions have appeared. Thanks to the powerful feature extraction and transfer learning capabilities of convolutional neural networks, such methods have achieved quite high levels of tracking performance and increasingly fast real-time performance, but the problem of reduced tracking accuracy caused by target appearance changes and inaccurate feature extraction still exists.

[0005] Some methods use uniform distribution sampling to let the target shift near the center point, which can alleviate the impact of the network due to the destruction of strict translation invariance, i.e. eliminating position bias, breaking the limitation of twin network for tracking that cannot effectively utilize deep network. At the same time, the traditional correlation operation is replaced by Depth-wise separated cross-correlation to obtain a score map with multiple channels and different semantic features, making the tracking target more accurate. However, this method presets the anchor box, which brings prior knowledge that does not conform to the characteristics of video target tracking, so the preset anchor box will limit the robustness and accuracy of the tracker. In addition, with the continuous development of attention mechanism algorithms, considering their simplicity and high performance, more and more target tracking algorithms are combined with attention mechanisms and show satisfactory results. However, this method does not fully consider the limitations of the target template in the twin network framework that cannot be updated online, and the independence of the features of the target template and the search image in the twin architecture. Therefore, the improvement of tracking accuracy is still not very obvious. SUMMARY

[0006] Therefore, the purpose of the present application is to provide a target tracking method based on multi-layer attention feature enhancement. In order to solve the problem of inaccurate feature extraction and similar target interference in the existing network model, which leads to inaccurate target tracking, the method proposes a multi-layer attention feature enhancement module to enhance the features of cross-correlation to improve the discriminative ability of the network. On this basis, in order to adapt to the target appearance change and similar object interference in the tracking process, the method proposes a template online updating and classifier learning strategy.

[0007] To achieve the above purpose, the present application provides the following technical solutions:

[0008] A target tracking method based on multi-layer attention feature enhancement, comprising the following steps:

[0009] S1: a target tracking network based on a twin network is constructed, a modified ResNet50 is used as a backbone network, and feature extraction is performed on a template picture and a search picture;

[0010] S2: a feature enhancement module is embedded in the last three layers in the backbone network to selectively enhance useful features, so that the features of the last three layers of ResNet50 interact more frequently;

[0011] S3: the template features and search features extracted by the backbone network are subjected to cross-correlation operation to obtain a response map;

[0012] S4: the cross-correlation feature map is classified and regressed to obtain the best tracking frame in multiple prediction frames, so as to realize tracking and online update of the template picture. An online template update and classifier learning strategy is adopted to adapt to the tracking drift problem caused by the continuous change of the appearance of the target in the tracking process, and the interference of similar backgrounds is largely resisted.

[0013] Further, the step S1 specifically comprises the following steps:

[0014] S11: pre-processing is performed on the input video sequence, if the video frame size is smaller than the preset size, the RGB average value is used for padding, if the video frame size is larger than the preset size, the edge is cropped, and the position of the bounding box label is corrected;

[0015] S12: the first frame picture in the video is used as a template branch, the template picture is cropped according to the target to be tracked, and the subsequent frames are used as a search branch;

[0016] S13: ResNet50 is used as a backbone network, and feature extraction is performed on the template branch image and the search branch image.

[0017] Further, the feature enhancement module comprises a self-attention module and an inter-attention module, and the step S2 specifically comprises the following steps:

[0018] S21: the self-attention module adaptively weights the response of each channel by calculating the attention feature map of the channel, and reduces the influence of the response of the irrelevant channel; the information of each position in space is captured by calculating the attention feature map of the spatial position, so that the feature of each position can capture the global information of the image;

[0019] S22: the inter-attention module first calculates the attention feature map according to the information of each branch, then transmits the feature map to another branch, and the branch receiving the feature map enhances the features extracted by itself according to the feature map, so as to realize more effective feature extraction;

[0020] S23: reference the self-attention module and the mutual-attention module to L3, L4 and L5 layers of the backbone network, so that the features of the input mutual correlation are interacted more frequently, and the feature expression ability is enhanced.

[0021] Further, the step S3 specifically comprises the following steps:

[0022] S31: performing a cross-correlation operation on the enhanced template features and the search features to obtain a response map;

[0023] S32: selecting a region with the maximum response value in the response map as the position of the tracking target.

[0024] Further, the step S31 of performing a cross-correlation operation on the enhanced template features and the search features to obtain a response map specifically comprises:

[0025] The template branch image is represented as z, the search branch image is represented as x, the deep convolutional network is a function f, and the feature extraction network performs the same φ transformation on the two inputs to obtain a similarity measurement function g:

[0026] f(z) = g(φ(z), φ(x))

[0027] Further, the step S4 specifically comprises the following steps:

[0028] S41: using a region proposal network RPN to realize classification and regression of the target, wherein the classification is used to distinguish foreground and background, the foreground refers to the position of the target, and the background refers to the position of the non-target; and the regression is used to determine the size of the target;

[0029] S42: performing weighted averaging on the obtained cross-correlation feature map to obtain a classification feature map and a bias coordinate map;

[0030] S43: converting the obtained bias coordinate map into a plurality of prediction boxes;

[0031] S44: applying a translation penalty and a scale penalty to obtain the best tracking box from the plurality of prediction boxes, so as to realize tracking;

[0032] S45: updating the target template in a timely manner, that is, in the tracking process, the tracking result of the algorithm on the historical frame is continuously transmitted into the target template in a certain way for tracking of the current frame.

[0033] The present application has the advantages that: the feature enhancement module is embedded in the last three layers of the backbone network, the feature interaction is more frequent, the feature expression ability is enhanced, and the tracker obtains more accurate target positioning information. In addition, an online template updating and classifier learning strategy is used to adapt to the tracking drift problem caused by the continuous change of the target appearance in the tracking process, and the interference of similar backgrounds is largely resisted.

[0034] Additional advantages, objects, and features of the application will be set forth in part by the description that follows, and in part will become apparent to those skilled in the art upon examination of same, or can be learned by practice of the application. The objects and other advantages of the application can be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0035] To make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0036] Figure 1 is a network model structure block diagram designed by the present application;

[0037] Figure 2 is a feature enhancement module structure diagram of the present application;

[0038] Figure 3 is a classifier structure diagram of the present application;

[0039] Figure 4 is a result comparison diagram of different algorithms on the OTB100 test data set. DETAILED DESCRIPTION

[0040] The advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and should not be understood as a limitation of the present application. In order to better illustrate the embodiments of the present application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the actual size of the product. It can be understood by those skilled in the art that some well-known structures and their descriptions in the drawings may be omitted.

[0042] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0043] The present application provides a target tracking method based on multi-layer attention feature enhancement, which is mainly divided into four parts, the first part is to construct a target tracking network based on a convolutional neural network, using a modified ResNet50 as the backbone network to extract features from the template picture and the search picture; the second part is to selectively enhance useful features by referring to a feature enhancement module, so that the interaction of the last three layers of ResNet50 is more frequent; the third part is to perform cross-correlation operation on the template features and search features extracted by the backbone network to obtain a response map; the fourth part proposes an online template updating and classifier learning strategy to adapt to the tracking drift problem caused by the continuous change of target appearance in the tracking process, which greatly resists the interference of similar backgrounds.

[0044] The network model structure of the target tracking method based on multi-layer attention feature enhancement provided by the present application is as shown in Figure 1 The feature enhancement module is embedded in the last 3 layers of the backbone network, so that the feature interaction is more frequent, the expression ability of the feature is enhanced, and the tracker obtains more accurate target positioning information. In addition, an online template updating and classifier learning strategy is used to adapt to the tracking drift problem caused by the continuous change of target appearance in the tracking process, which greatly resists the interference of similar backgrounds. Specifically, the following steps are included:

[0045] Step 1: Construct a target tracking network based on a twin network, use a modified ResNet50 as the backbone network to extract features from the template picture and the search picture;

[0046] (1) Preprocess the input video sequence, if the video frame size is less than the preset size, use the average value of RGB to fill, if the size is greater than the preset size, crop the edge, and at the same time, correct the position of the bounding box label;

[0047] (2) Take the first frame picture in the video as the template branch, crop the template picture according to the target to be tracked, and take the subsequent frames as the search branch;

[0048] (3) Using ResNet50 as the backbone network, the template branch image and the search branch image are subjected to feature extraction;

[0049] Step 2: The feature enhancement module is cited to selectively enhance useful features, so that the last three layers of ResNet50 interact more frequently; the structure of the feature enhancement module of the application is shown in Figure 2 The feature enhancement module includes a self-attention module and an inter-attention module. The self-attention feature learns rich image context information in the spatial domain and selectively enhances the mutual dependence between channel features in the channel domain; the inter-attention feature aggregates and communicates rich information between the template and the search region, improving the discrimination ability of the feature.

[0050] (1) The self-attention module adaptively weights the response of each channel by calculating the channel attention feature map, reducing the influence of the response of irrelevant channels; the information of each position in space is captured by calculating the spatial position attention feature map, so that the feature of each position can capture the global information of the image;

[0051] (2) The inter-attention module first calculates the attention feature map according to the information of each branch, and then transmits this feature map to another branch. The branch receiving the feature map enhances the feature extracted by itself according to the feature map, so as to realize more effective feature extraction;

[0052] (3) The feature enhancement module is cited in the backbone network, so that the features related to the input interact more frequently, which is conducive to the enhancement of feature expression ability.

[0053] Step 3: The template feature and the search feature extracted by the backbone network are subjected to cross-correlation operation to obtain a response map;

[0054] (1) The enhanced template feature and the search feature are subjected to cross-correlation operation to obtain a response map;

[0055] The template branch image is represented as z, the search branch image is represented as x, the deep convolutional network is used as the function f, and the feature extraction network is subjected to the same φ transformation on the two inputs to obtain the similarity measure function g:

[0056] f(z)=g(φ(z),φ(x))

[0057] (2) In the response map, the region with the maximum response value is selected as the position of the tracking target.

[0058] Step 4: An online template updating and classifier learning strategy is adopted to adapt to the tracking drift problem caused by the continuous change of the appearance of the target in the tracking process, which greatly resists the interference of similar backgrounds.

[0059] (1) The classifier learning strategy in the application is shown in Figure 3 As shown in the figure, the region proposal network (RPN) is used to realize classification and regression of the target, wherein the classification is to distinguish the foreground and the background, the foreground refers to the position of the target, and the background refers to the position of the non-target.

[0060] The regression is to determine the size of the target;

[0061] (2) The obtained cross-correlation feature map is weighted and averaged to obtain a classification feature map and a deviation coordinate map;

[0062] (3) The obtained deviation coordinate map is converted into a plurality of prediction boxes;

[0063] (4) Translation penalty and scale penalty are applied to obtain the best tracking box from the plurality of prediction boxes, so as to realize tracking;

[0064] (5) The target template is updated in time, that is, in the tracking process, the tracking result of the algorithm on the historical frame is continuously transmitted into the target template in a certain way for tracking of the current frame.

[0065] Step 5: Determination of tracking result

[0066] Figure 4 The tracking result schematic diagram of the application is shown in the figure. The template branch feature and the search branch feature extracted by the optimized backbone network are cross-correlated, and the tracking result is obtained.

[0067] The target tracking method based on multi-layer attention feature enhancement designed by the application mainly includes two stages of training and testing.

[0068] 1) Training stage

[0069] The training stage mainly includes feature extraction and updating of model weight parameters. The model is trained by using the preprocessed video sequence, the values predicted by the model are compared with the label true values to calculate the loss values, the weight parameters of the model are updated according to the total loss value and using the gradient back propagation algorithm. When the iteration number of the model reaches the preset value, the training process is terminated and the weight parameters are saved.

[0070] 2) Testing stage

[0071] The testing stage needs to load the trained model weight parameters, and only uses the scaling method to make the size of the input image frame meet the input requirements of the model. At this time, the model no longer performs gradient back propagation but directly outputs the tracking result to realize tracking of the target in the video.

[0072] Finally, it is to be explained that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the purpose and scope of the technical solutions, and all should be covered in the scope of the claims of the present application.

Claims

1. A target tracking method based on multi-layer attention feature enhancement, characterized in that: Comprise the following steps: S1: construct a target tracking network based on a twin network, use a modified ResNet50 as a backbone network, and perform feature extraction on a template picture and a search picture; S2: embedding a feature enhancement module in the last three layers of the backbone network to selectively enhance useful features; the feature enhancement module comprises a self-attention module and an inter-attention module, and the step S2 specifically comprises the following steps: S21: the self-attention module adaptively weights the response of each channel by calculating the attention feature map of the channel, and reduces the influence of the response of the irrelevant channel; the information of each position in space is captured by calculating the attention feature map of the spatial position, so that the feature of each position can capture the global information of the image; S22: the inter-attention module first calculates an attention feature map according to the information of each branch itself, and then transmits the feature map to another branch; the branch receiving the feature map enhances the feature extracted by itself according to the feature map; S23: the self-attention module and the inter-attention module are referred to the L3, L4 and L5 layers of the backbone network; S3: performing cross-correlation operation on the template features and search features extracted by the backbone network to obtain a response map; S4: classifying and regressing the cross-correlation feature map to obtain the best tracking frame among multiple prediction frames, thereby realizing tracking, and updating the template picture online; step S4 specifically comprises the following steps: S41: using a region proposal network RPN to realize classification and regression of the target, wherein the classification is used to distinguish foreground and background, and the foreground refers to the position of the target, and the background refers to the position of the non-target; the regression is used to determine the size of the target; S42: performing weighted average on the obtained cross-correlation feature map to obtain a classification feature map and a deviation coordinate map; S43: converting the obtained deviation coordinate map into multiple prediction frames; S44: applying translation penalty and scale penalty to obtain the best tracking frame from the multiple prediction frames, thereby realizing tracking; S45: timely updating the target template, that is, during the tracking process, the tracking results of the algorithm on the historical frames are continuously transmitted into the target template for tracking of the current frame.

2. The target tracking method based on multi-layer attention feature enhancement according to claim 1, characterized in that: The step S1 specifically comprises the following steps: S11: pre-processing the input video sequence, if the video frame size is less than the preset size, using the RGB average value to fill, if the video frame size is greater than the preset size, cutting the edge, and simultaneously correcting the position of the bounding box label; S12: taking the first frame picture in the video as a template branch, and cutting the template picture according to the target to be tracked, and taking the subsequent frames as search branches; S13: using ResNet50 as a backbone network to perform feature extraction on the template branch image and the search branch image.

3. The target tracking method based on multi-layer attention feature enhancement according to claim 1, characterized in that: The step S3 specifically comprises the following steps: S31: performing cross-correlation operation on the enhanced template features and search features to obtain a response map; S32: selecting the region with the maximum response value in the response map as the position of the tracking target.

4. The target tracking method based on multi-layer attention feature enhancement according to claim 3, characterized in that: The step S31 of performing cross-correlation operation on the enhanced template features and search features to obtain a response map is specifically: Template branch image representation is z , search branch image representation is x , deep convolutional network as f function, feature extraction network on two inputs the same φ transform, get similarity measure function g : f ( z ) = g ( φ ( z ) , φ ( x ) )。

Images