Visual target tracking method and device based on attention alignment

Abstract

The invention discloses a visual target tracking method and device based on attention alignment, and belongs to the field of visual target tracking, and the method comprises the steps: firstly, carrying out the bidirectional mutual query of a template and search region information, carrying out the calculation to obtain respective attention matrixes for target information, then aligning the two attention matrixes through an element-by-element addition mode, and carrying out the tracking of a visual target. Focusing attention on a real target and inhibiting false attention caused by interferents to obtain an attention matrix of corrected accurate focusing target information; based on the corrected attention matrix, information most relevant to the current target is extracted from the template and the search area, and accurate target representation is obtained; and finally, identifying and positioning a target needing to be tracked in the video based on the accurate target representation. The method solves the problems that a tracking method is disturbed and confused, and tracking drifts and even errors exist, and improves the accuracy and robustness of target tracking in a complex scene.

Description

Technical Field

[0001] The present invention belongs to the field of visual object tracking, and particularly relates to a visual object tracking method and device based on attention alignment. Background Art

[0002] Visual object tracking is an important research field in computer vision, aiming to locate the target in subsequent frames based on the target template of a given video using a tracking method. Visual object tracking technology is widely applied and involves all aspects of people's production and life, such as intelligent driving, video surveillance analysis, human-computer interaction, etc.

[0003] Focusing on capturing important target information in visual object tracking is crucial for achieving stable tracking. However, the real scene is complex and changeable, the target state changes at any time during the tracking process, and there are similar distractors in the scene. These factors may mislead the tracker at any time. Therefore, maintaining accurate recognition and positioning of the target in consecutive frames is still an urgent problem to be solved in the general visual tracking task.

[0004] The existing visual object tracking methods usually refer to the target representation of a given initial template frame, and identify the most similar object in the search area as the target, that is, using the search area information as a target query to capture the most similar information in the template as the target representation. However, the existing visual object tracking methods ignore a key problem: there are not only similarities between the target in the template and the search area, but also many distractors that are similar in appearance to the target. These distractors will distract the tracker's attention during the process of target representation learning. Therefore, it is difficult for the tracker to learn an accurate target representation, and the inaccurate target representation will deceive the tracker, resulting in problems such as tracking drift and errors. The technical problem to be solved by the present invention is that when the existing technology performs target representation learning in the interactive template and search area, the distractor objects in the search area will distract the tracker's attention, resulting in inaccurate target representation. Summary of the Invention

[0005] To solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A visual object tracking method based on attention alignment, comprising:

[0007] Step 1: Obtain a template image and a search area image ;

[0008] Step 2: Construct an integrated network for feature extraction and interaction, and a prediction head including classification and regression networks; the integrated network for feature extraction and interaction includes convolutional blocks and a focused object attention mechanism for interacting and extracting object information in the template and search regions; the prediction head including classification and regression networks is used to identify and locate the objects in the search regions;

[0009] Among them, the specific interaction process between the template and the search region in the integrated network for feature extraction and interaction is as follows:

[0010] Step 2.1: Load the pre-trained weights and parameter configurations of the network, and use the cropped template image and the search region image as the network input;

[0011] Step 2.2: Use convolutional blocks to separately transform the input template image and the search region image into token embeddings and ;

[0012] Step 2.3: Then use the token embeddings and as the input of the integrated network for feature extraction and interaction, and use the proposed focused object attention mechanism to interact and for object representation learning;

[0013] Step 2.4: Based on the query, key, and value features of the template and the search region, achieve the attention interaction of the template and search region features and learn accurate object representations;

[0014] Step 2.5: The prediction head including classification and regression networks receives the object representation output by the integrated network for feature extraction and interaction as the input.

[0015] A visual object tracking device based on attention alignment includes:

[0016] Template and search region image acquisition module: Obtain the template image and the search region image from a video sequence;

[0017] Integrated network for feature extraction and interaction, prediction head construction module: Construct an integrated network for feature extraction and interaction, and a prediction head including classification and regression networks; the integrated network for feature extraction and interaction includes convolutional blocks and a focused object attention mechanism for interacting and extracting object information in the template and search regions; the prediction head including classification and regression networks is used to identify and locate the objects in the search regions;

[0018] Among them, the specific process of the interaction between the template and the search area in the feature extraction and interaction integrated network is as follows:

[0019] Load the pre-trained weights and parameter configurations of the network, and use the cropped template image and the search area image as the network input;

[0020] Use convolutional blocks to separately transform the input template image and the search area image into token embeddings and ;

[0021] Then, use the token embeddings and as the input of the feature extraction and interaction integrated network, and use the proposed focused object attention mechanism to interact and for object representation learning;

[0022] Based on the query, key, and value features of the template and the search area, achieve attention interaction between the template and search area features and learn accurate object representations;

[0023] The prediction head containing the classification and regression networks receives the object representation output by the feature extraction and interaction integrated network

[0024] An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the above-mentioned visual object tracking method based on attention alignment.

[0025] A non-transitory computer-readable storage medium stores a computer program, which when executed by a processor implements the steps of the above-mentioned visual object tracking method based on attention alignment.

[0026] The present invention has the following beneficial effects:

[0027] The present invention proposes a visual object tracking method based on attention alignment, aiming to improve the accuracy and robustness of object tracking in complex scenarios. Its core lies in achieving precise object representation learning through a focused object attention mechanism and an attention alignment strategy. The specific process is as follows: First, extract the template and search region images from the video sequence, and convert them into token embeddings using convolutional blocks. Subsequently, map the token embeddings to query, key, and value features through a fully connected layer, and calculate the attention weight matrix using bidirectional interaction. Correct the attention matrix of the search region through the attention alignment method to suppress interference information, thereby obtaining an accurate object representation. Finally, input the object representation into the prediction head, and use the classification and regression networks to respectively determine whether the tracker correctly identifies the object and predict the bounding box of the object. This method effectively improves the tracking accuracy and anti-interference ability of the tracker in complex scenarios through the focused object attention mechanism and the attention alignment strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of the visual object tracking method based on attention alignment of the present invention;

[0029] Figure 2 is a schematic diagram of the focused object attention mechanism of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0030] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0031] The present invention provides a visual object tracking method based on attention alignment. The overall process of this method is: After obtaining the template and search region images from a video sequence and first, use convolutional blocks to convert the template and search region images into token embeddings respectively and , then use the token embeddings and as the input of the feature extraction and interaction integrated network, and use the proposed focused object attention mechanism to interact and for object representation learning. Among them, use the proposed focused object attention mechanism to interact and To perform target representation learning specifically: In the proposed focused target attention mechanism, the token embeddings of the template and the search region are mapped through a fully connected layer to obtain the template query feature, the template key feature, the template value feature, the search region query feature, the search region key feature, and the search region value feature. Then, the template query feature and the search region query feature are respectively interacted bidirectionally with the transposed key features of each other to calculate the attention weight matrices of the template and the search region for the target information respectively. and ; Subsequently, use the attention alignment method to align and , focus the tracker's attention on the true target information, and suppress the false attention caused by interference information to obtain the corrected search region attention matrix ; Based on capture the most relevant target information from the value features of the template and the search region to obtain an accurate target representation ; Finally, send the accurate target representation into the prediction head containing the classification and regression networks, use the regression network to predict the bounding box of the target, and use the classification network to determine whether the tracker correctly identifies the target.

[0032] As Figure 1 shown, the steps of the present invention are specifically as follows:

[0033] Step 1: Obtain a video sequence, crop the first frame as the template image , and use each subsequent frame as the search region image .

[0034] Step 2: Construct an integrated network for feature extraction and interaction, and a prediction head containing classification and regression networks. The integrated network for feature extraction and interaction includes a convolutional block and a focused target attention mechanism, which are used to interact and extract the target information in the template and the search region; the prediction head containing classification and regression networks is used to identify and locate the target in the search region.

[0035] The specific process of the interaction between the template and the search region in the integrated network for feature extraction and interaction in Step 2 is as follows:

[0036] Step 2.1: Load the pre-trained weights and parameter configurations of the network, and use the cropped template image and the search region image as the network input;

[0037] Step 2.2: Use the convolutional block to respectively transform the input template image and the search region image into token embeddings and ;

[0038] Step 2.3: Then embed the token and as the input of the integrated feature extraction and interaction network, and utilize the proposed focused object attention mechanism to interact and for object representation learning;

[0039] Utilize the fully connected layer in the focused object attention mechanism to map and to obtain the query, key, and value features of the template and the search region: template query feature, template key feature, template value feature, search region query feature, search region key feature, and search region value feature.

[0040] Step 2.4: As Figure 2 shown, based on the query, key, and value features of the template and the search region, implement the attention interaction between the template and the search region features and learn an accurate object representation. The specific operation process is as follows:

[0041] Step 2.4.1: Perform attention interaction between the query feature of the template and the transposed key feature of the search region, and calculate the attention weight matrix of the template to the object information ;

[0042] Step 2.4.2: Perform attention interaction between the query feature of the search region and the transposed key feature of the template, and calculate the attention weight matrix of the search region to the object information ;

[0043] Step 2.4.3: After performing the transpose operation on and aligning the spatial dimensions with , utilize the element-wise addition method to align the attention foci of and , suppress the false attention caused by interference information, and make the attention of the integrated feature extraction and interaction network accurately focus on the object information, obtaining the corrected attention matrix of the search region ;

[0044] Step 2.4.4: Based on the corrected attention matrix of the search region capture the most relevant object information from the value features of the template and the search region to obtain an accurate object representation .

[0045] Step 2.5: The prediction head containing the classification and regression networks receives the object representation output by the integrated feature extraction and interaction network as the input. Among them, the regression network predicts the object bounding box based on this object representation, and the classification network determines whether the tracker correctly identifies the object.

[0046] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, apparatus, system, or computer program product. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The solutions in the embodiments of the present invention can be implemented in various computer languages. For example, object-oriented programming languages such as Java and interpreted scripting languages such as JavaScript, etc.

[0047] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0048] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0049] These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process. Thus, the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0050] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present invention.

Claims

1. A visual target tracking method based on attention alignment, characterized in that: include: Step 1: Get a template image from a video sequence and search area image ; Step 2: construct a feature extraction and interaction integrated network and a prediction head including a classification and regression network; the feature extraction and interaction integrated network includes a convolution block and a focus target attention mechanism for interacting and extracting target information in the template and the search area; the prediction head including the classification and regression network is used to identify and locate the target in the search area; Among them, the specific process of interaction between template and search area in the feature extraction and interaction integrated network is as follows: Step 2.1: Load the network's pre-trained weights and parameter configuration and crop the template image and search area image As network input; Step 2.2: Use convolution blocks to transform the input template images separately and search area image Embed for token and ; Step 2.3: Then embed the token and As the input of the feature extraction and interaction integrated network, the proposed focused target attention mechanism is used to interact and To perform target representation learning; Step 2.4: Based on the query, key and value features of the template and search area, realize the attention interaction of the template and search area features and learn accurate target representation; Step 2.5: The prediction head containing the classification and regression networks receives the target representation output by the feature extraction and interactive integration network as input.

2. The visual target tracking method based on attention alignment according to claim 1, characterized in that: Step 1 includes: obtaining a video sequence and cropping the first frame as a template image , each subsequent frame is used as the search area image .

3. The visual target tracking method based on attention alignment according to claim 1, characterized in that: In step 2.3, the proposed focus target attention mechanism is used to interact and To learn the target representation, we use the fully connected layer in the focus target attention mechanism to and Mapping,gets the query, key and value features of the template and search,area: template query feature, template key feature, template value feature,,search area query feature, search area key feature and search area value feature.

4. The visual target tracking method based on attention alignment according to claim 1, characterized in that: The query, key and value features of the template and search area include: template query feature, template key feature, template value feature, search area query feature, search area key feature and search area value feature.

5. The visual target tracking method based on attention alignment according to claim 1, characterized in that: Step 2.4 specifically includes: Step 2.4.1: The query feature of the template and the transposed search area key feature interact with each other to calculate the attention weight matrix of the template to the target information ; Step 2.4.2: The query features of the search area and the transposed template key features interact with each other to calculate the attention weight matrix of the search area for the target information. ; Step 2.4.3: Align using attention alignment method and , focus the tracker's attention on the real target information and suppress the false attention caused by interference information, and obtain the corrected search area attention matrix ; Step 2.4.4: Attention matrix based on the corrected search area Capture the most relevant target information from the value features of the template and search area to obtain accurate target representation .

6. The visual target tracking method based on attention alignment according to claim 1, characterized in that: Step 2.4.3 specifically includes: After performing the transpose operation and Align the spatial dimensions by adding elements one by one and The attention focus of the search area is obtained by suppressing the false attention caused by interference information, so that the attention of the feature extraction and interaction integrated network is accurately focused on the target information, and the attention matrix of the search area after correction is obtained. .

7. The visual target tracking method based on attention alignment according to claim 1, characterized in that: In step 2.5, the regression network predicts the target bounding box based on the target representation, and the classification network determines whether the tracker correctly identifies the target.

8. A visual target tracking device based on attention alignment, characterized in that: include: Template and search area image acquisition module: Get a template image from a video sequence and search area image ; Feature extraction and interaction integrated network, prediction head construction module: construct a feature extraction and interaction integrated network, a prediction head including a classification and regression network; the feature extraction and interaction integrated network includes a convolution block and a focus target attention mechanism for interacting and extracting target information in the template and search area; the prediction head including the classification and regression network is used to identify and locate the target in the search area; Among them, the specific process of interaction between template and search area in the feature extraction and interaction integrated network is as follows: Load the network's pre-trained weights and parameter configuration and crop the template image and search area image As network input; Use convolution blocks to transform the input template images separately and search area image Embed for token and ; Then embed the token into and As the input of the feature extraction and interaction integrated network, the proposed focused target attention mechanism is used to interact and To perform target representation learning; Based on the query, key and value features of the template and search area, the attention interaction of the template and search area features is realized and accurate target representation is learned; The prediction head, which consists of a classification and regression network, receives the target representation output by the feature extraction and interactive integration network. as input.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the visual target tracking method based on attention alignment as described in any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual target tracking method based on attention alignment as described in any one of claims 1 to 7 are implemented.

Images