Lightweight target tracking system based on mixed linear attention
By introducing a hybrid linear attention block and feature adaptive network into the target tracking system, conflicts between classification and regression tasks are resolved, tracking accuracy and discriminative capabilities are improved, and suitable for edge devices.
Patent Information
- Application Number
- CN202411508567.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-25
AI Technical Summary
In the existing target tracking system, the classification and regression tasks cause task conflicts due to different characteristics, resulting in negative migration, which affects the tracking effect.
A lightweight target tracking system based on hybrid linear attention is adopted to decouple the features of the input prediction head through feature adaptive networks to adapt to the needs of different tasks, and to improve the discriminantity of the algorithm in feature extraction and correlation modeling through hybrid linear attention blocks.
It effectively solves the problem of task conflict, improves the accuracy and discriminant ability of target tracking, and does not increase in reasoning delay, and is suitable for edge devices.
Smart Images
Figure CN119515917A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of target tracking, and in particular to a lightweight target tracking system based on hybrid linear attention. Background Art
[0002] Object tracking refers to locating the target in subsequent frames given the position of the target in the first frame of the video. Object tracking is a basic but difficult problem in computer vision. In recent years, due to its super global feature learning ability, Transformer has been widely used in the field of single target tracking. These Transformer-based trackers can well model the relationship between the tracking target and the template using Transformer's self-attention and mutual attention. However, the calculation of attention is proportional to the square of the length of the input sequence, which makes these trackers usually have a large amount of calculation, resulting in the model needing to run on some specific GPU devices, but cannot be applied on some edge or mobile devices.
[0003] The current mainstream tracker framework consists of three parts: feature extraction, correlation modeling, and prediction head, such as Figure 1 As shown in the figure, some methods perform feature extraction and related modeling simultaneously, so that better target-related features can be extracted. Single target tracking divides positioning into two sub-problems: classification and regression. Therefore, the prediction head contains a classification head and a regression head. The classification head is used to identify the tracking target from the background, and the regression head is used to predict the coordinates of the corner points of the tracking target. However, for the two tasks of classification and regression, they have different requirements for features. Classification requires that the features have translation and scale invariance. No matter where the target is in the picture, what category it is, the learned features will not change due to transformations such as scale, rotation and position. Positioning is a task that is sensitive to position information. The learned features are definitely different when the target is located at different positions and scales. The area of interest for classification is the most prominent area of the object, that is, the most discriminative place; while the area of interest for the localization task is the edge area of the object. In short, the response areas of these two tasks are different for features. However, in the existing target tracking framework, they usually input the same backbone network features into the classification head and the regression head at the same time, which causes the problem of task conflict. In this way, the joint training of the classification head and the regression head will lead to negative transfer between the conflicting tasks, which will eventually affect the tracking effect.
[0004] In order to solve the above problems, the present invention proposes a lightweight target tracking system based on hybrid linear attention. Summary of the invention
[0005] 1. Technical Problems to be Solved by the Present Invention
[0006] The purpose of the present invention is to propose a lightweight target tracking system based on hybrid linear attention to solve the task conflict problem raised in the background technology; it also innovatively proposes a feature adaptive network based on low-rank adaptation, which can adapt to the characteristics of the task according to task learning, thereby decoupling the characteristics of the input prediction head to solve the task conflict problem without increasing the inference delay.
[0007] 2. Technical solution
[0008] To achieve the above object, the present invention provides the following technical solutions:
[0009] A lightweight object tracking system based on hybrid linear attention, including feature extraction network, feature adaptation network and prediction head;
[0010] The feature extraction network includes a weight sharing module and an interaction module; the weight sharing module consists of a convolution and three inverted residual blocks, which are used to extract local features of the template and the search image and downsample the image; the interaction module consists of an inverted residual block and a hybrid linear attention block cascaded twice, which is used to perform correlation modeling on the template and the search, so as to extract features related to the template in the search image and improve the discriminability of the tracking algorithm;
[0011] The feature adaptation network is composed of feature adaptive convolution and activation function cascade, which is used to correct the features of the input prediction head so that the features of the input prediction head are adapted to the tasks of different prediction heads;
[0012] The prediction head consists of a classification prediction head and a regression prediction head. The classification prediction head consists of two mixed linear attention block cascades, and the regression prediction head consists of four mixed linear attention block cascades.
[0013] Preferably, the input of the feature extraction network is a template image and a search image, and the processing flow of the feature extraction network specifically includes the following contents:
[0014] A1. Reduce the spatial size of the template image and the search image through a 3×3 convolution, and then extract local features and downsample the image through three inverted residual blocks;
[0015] A2: Send the template image features and search image features processed by A1 to the inverted residual block and the hybrid linear attention block, and encode the local spatial information and reduce the number of channels through a depth-separable 3×3 convolution and a conventional 1×1 convolution;
[0016] A3. Expand the template image features and the search image features to generate a sequence and concatenate them together to perform self-attention calculation with linear complexity to extract contextual information on the target and perform related modeling;
[0017] A4. Use three 1×1 convolutions to get the query, key, and value respectively, then perform a normalized exponential operation on the query to get a context score, which represents the target and context information; perform a weighted operation on the context score and the key to get a context vector, which is then propagated to the value of the linear rectifier activation through a broadcast element-wise multiplication operation, and the output is passed through a linear layer to get the final attention output;
[0018] A5, the attention output obtained in A4 is divided into two parts and folded to obtain the global features of the template and the global features of the search area after interaction, and then the global features of the template and the global features of the search area are spliced together with the local features and sent to a convolutional layer to fuse the local and global features; then the global features of the template and the global features of the search area are spliced together with the template features and the search area features of the input attention block as their respective outputs to increase the network optimization capability;
[0019] A6. Use a pixel-by-pixel correlation operation to fuse the template features and the search area features to obtain a response map.
[0020] Preferably, the feature adaptive network further includes the following contents:
[0021] A task-adaptive convolution is proposed, in which a common weight matrix, a classification weight matrix and a regression weight matrix are inserted in parallel next to the weight matrix of the convolutional neural network layer; the common weight represents a common feature subspace, which is used to learn common features between different tasks to ensure basic commonality between various tasks; the classification weight matrix represents a classification feature subspace, which is used to capture scale-invariant features and contextual information; the regression weight matrix represents a regression feature subspace, which is used to learn spatial detail information to facilitate regression calculation;
[0022] Based on the task adaptive convolution, a feature adaptive network is proposed. The feature adaptive network concatenates the search area features and the response feature map as its input, then reduces the number of channels through a task adaptive convolution and then performs an average pooling operation on its spatial dimension; then, a task adaptive convolution and a linear rectification activation function are used to keep the number of channels unchanged; and then a task adaptive convolution and a logical activation function are used to learn and capture important channel information of different tasks, and finally a task adaptive convolution is used to obtain the final output.
[0023] Preferably, the input of the classification prediction head is the output of the feature adaptation network spliced together without including the regression feature subspace; the input of the regression prediction head is the output of the feature adaptation network spliced together without including the classification feature subspace.
[0024] 3. Beneficial effects
[0025] (2) The present invention proposes a target tracking system based on CNN and Transformer with linear attention (denoted as ECTTrack), which combines the advantages of CNN and Transformer. It has the spatial inductive bias and correlation between local pixels like CNN, and also has the global modeling ability of Transformer. In addition, the tracking backbone network performs feature extraction and correlation modeling simultaneously, which can extract more features related to the tracking target in the search image, thereby improving the algorithm's discrimination ability.
[0026] (3) The present invention also proposes a feature adaptive network based on low-rank adaptation to solve the task conflict problem. It can adjust the learned features according to the task to adapt them to the prediction heads of different tasks, and it will not increase the inference delay, thereby ensuring the inference speed of the entire tracker. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 It is a framework diagram of the mainstream tracking algorithm proposed in the background technology of the present invention;
[0028] Figure 2 The overall framework diagram of the lightweight target tracking system based on hybrid linear attention proposed in Example 1 of the present invention;
[0029] Figure 3 This is a diagram of the efficient hybrid linear attention block structure proposed in Example 1 of the present invention;
[0030] Figure 4 It is a linear self-attention structure diagram proposed in Example 1 of the present invention;
[0031] Figure 5 It is a structural diagram of the task adaptive convolution and feature adaptive network proposed in Example 1 of the present invention;
[0032] Figure 6 This is a visualization diagram of the tracking results proposed in Example 3 of the present invention. DETAILED DESCRIPTION
[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.
[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.
[0036] Embodiment 1:
[0037] This paper proposes a lightweight target tracking system based on hybrid linear attention (called ECTTrack), which is designed based on CNN and Transformer and mainly consists of three parts: feature extraction network, feature adaptation network and prediction head. The specific design is as follows Figure 2 As shown, including the following:
[0038] (1) Feature extraction network
[0039] The input of the feature extraction network is a template image Z and a search image X. First, a 3×3 convolution is used to reduce the spatial size of the template image and the search image. Then, three inverted residual blocks are used to extract local features and downsample them. The weights of the above 3×3 convolution and inverted residual blocks are shared by the template branch and the search image branch. Then, the template and search image features are sent to the inverted residual block and the efficient hybrid linear attention block. The structure of the efficient hybrid linear attention block is as follows: Figure 3 As shown. The present invention uses a depth-separable 3×3 convolution and a conventional 1×1 convolution to encode local spatial information and reduce the number of channels. In order to extract the context information of the target and perform related modeling at the same time, the present invention expands the template feature and the search feature to generate a sequence and splices them together to perform self-attention calculation with linear complexity, as shown in Figure 4 As shown. The present invention first uses three 1×1 convolutions to obtain the query, key and value respectively. Then the query is normalized and indexed to obtain the context score, which represents the target and context information. This context score is weighted with the key to obtain the context vector, and then the context vector is propagated to the value of the linear rectifier activation through a broadcast element multiplication operation, and the output is passed through a linear layer to obtain the final attention output. Then the attention output is divided into two parts and folded to obtain the interactive template global features and search area global features respectively, and then they are spliced together with the local features and sent to a convolution layer for fusing local and global features. Finally, in order to increase the network optimization capability, the present invention splices them together with the template features and search area features of the input attention block as their respective outputs. Finally, the present invention uses a pixel-by-pixel correlation operation to fuse the template features and the search area features to obtain a response map.
[0040] (2) Feature Adaptive Network Based on Low-Rank Adaptation
[0041] In deep learning, a matrix is usually used To represent the weight of the convolutional neural network, assuming that the input is x, the calculation of the convolutional neural network can be expressed by formula (1).
[0042] Y=W·x(1)
[0043] The present invention proposes a task-adaptive convolution, in which three weight matrices in parallel are inserted next to the matrix weight. Due to the low rank of the matrix, each weight matrix is split into a reduced-dimensional matrix and an increased-dimensional matrix, such as Figure 5 As shown, the calculation of task adaptive convolution can be expressed by formula (2). During training, the original weight W is fixed, and only the dimension reduction matrix and dimension increase matrix are trained. In this way, training can be completed with extremely small parameters. During testing, the dimension reduction matrix and dimension increase matrix are added to the original weight, so that no additional delay will be generated during inference.
[0044] Y i =(W+ΔW i )·x(2)
[0045] Task-adaptive convolution divides feature learning into three different subspaces, and learns different features respectively to adapt them to different tasks of the prediction head. Specifically, the present invention inserts three different weight matrices of specific tasks in parallel next to the weight matrix of the convolutional neural network layer. The weight matrices of these three specific tasks are a common weight matrix, a classification weight matrix, and a regression weight matrix. The common weights represent the common feature subspace, which mainly learns some common features between different tasks to ensure basic commonality between various tasks. The classification weight matrix represents the classification feature subspace, which mainly captures scale-invariant features and contextual information. The regression weight matrix represents the regression feature subspace, which mainly learns spatial detail information to facilitate regression calculations.
[0046] Based on this task-adaptive convolution, this aspect proposes a feature adaptive network, which concatenates the search area features and the response feature map as its input, and then reduces the number of channels through a task-adaptive convolution and performs an average pooling operation on its spatial dimension, and then keeps the number of channels unchanged through a task-adaptive convolution and a linear rectifier activation function. It is followed by a task-adaptive convolution and a logical activation function to learn and capture important channel information of different tasks, and finally a task-adaptive convolution is performed to obtain the final output, as shown in the figure. After adjustment by the feature adaptive network, the features of the input prediction head correspond to the tasks of the prediction head one by one. The feature adaptive network ensures that the model can efficiently process the shared features related to multiple tasks while meeting the specific needs of individual tasks.
[0047] (3) Prediction Head
[0048] The prediction head proposed in the present invention mainly includes a classification prediction head and a regression prediction head. The input of the classification prediction head is the output of the feature adaptive network spliced together except for the regression feature subspace, and the input of the regression prediction head is the output of the feature adaptive network spliced together except for the classification feature subspace. The two heads use a 3×3 convolution to reduce the number of channels, and then the classification head cascades two efficient hybrid linear attention blocks, and the regression head cascades four efficient hybrid linear attention blocks.
[0049] Embodiment 2:
[0050] The performance of the target tracking system (ECTTrack) proposed in this invention is compared with some excellent lightweight tracking algorithms on the LaSOT, TrackingNet and GOT-10K datasets. The experimental results are shown in Table 1.
[0051] Table 1
[0052]
[0053] According to Table 1, the target tracking system (ECTTrack) proposed in the present invention has good tracking accuracy, few parameters, and can be run on CPU. The present invention conducts experiments on multiple challenging tracking datasets to verify the tracking performance. ECTTrack exceeds the current optimal tracking algorithm on all three datasets.
[0054] Embodiment 3:
[0055] In order to more intuitively observe the effect of the target tracking system (ECTTrack) proposed in the present invention, the present invention visualizes the tracking results of ECTTrack and other excellent trackers. The visualization results are as follows: Figure 6 As shown, the video in the fourth row is sheep_3 of lasot. When the tracking target is disturbed by a similar target, some algorithms experience tracking drift or even complete tracking failure. The ECTTrack proposed in the present invention successfully locates the target position, demonstrating a robust tracking effect.
[0056] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the protection scope of the present invention.
Claims
1. A lightweight target tracking system based on hybrid linear attention, characterized in that Includes feature extraction network, feature adaptation network and prediction head; The feature extraction network includes a weight sharing module and an interaction module; the weight sharing module consists of a convolution and three inverted residual blocks, which are used to extract local features of the template and the search image and downsample the image; the interaction module consists of an inverted residual block and a hybrid linear attention block cascaded twice, which is used to perform correlation modeling on the template and the search, so as to extract features related to the template in the search image and improve the discriminability of the tracking algorithm; The feature adaptation network is composed of feature adaptive convolution and activation function cascade, which is used to correct the features of the input prediction head so that the features of the input prediction head are adapted to the tasks of different prediction heads; The prediction head consists of a classification prediction head and a regression prediction head. The classification prediction head consists of two mixed linear attention block cascades, and the regression prediction head consists of four mixed linear attention block cascades.
2. A lightweight target tracking system based on hybrid linear attention according to claim 1, characterized in that: The input of the feature extraction network is a template image and a search image. The processing flow of the feature extraction network specifically includes the following contents: A1. Reduce the spatial size of the template image and the search image through a 3×3 convolution, and then extract local features and downsample the image through three inverted residual blocks; A2: Send the template image features and search image features processed by A1 to the inverted residual block and the hybrid linear attention block, and encode the local spatial information and reduce the number of channels through a depth-separable 3×3 convolution and a conventional 1×1 convolution; A3. Expand the template image features and the search image features to generate a sequence and concatenate them together to perform self-attention calculation with linear complexity to extract contextual information on the target and perform related modeling; A4. Use three 1×1 convolutions to obtain the query, key, and value respectively, and then perform a normalized index operation on the query to obtain a context score, which represents the target and context information; The context score is weighted with the key to obtain a context vector, which is then propagated to the value of the linear rectifier activation through a broadcast element-wise multiplication operation, and the output is passed through a linear layer to obtain the final attention output; A5, the attention output obtained in A4 is divided into two parts and folded to obtain the global features of the template and the global features of the search area after interaction, and then the global features of the template and the global features of the search area are spliced together with the local features and sent to a convolutional layer to fuse the local and global features; then the global features of the template and the global features of the search area are spliced together with the template features and the search area features of the input attention block as their respective outputs to increase the network optimization capability; A6. Use a pixel-by-pixel correlation operation to fuse the template features and the search area features to obtain a response map.
3. A lightweight target tracking system based on hybrid linear attention according to claim 1, characterized in that: The feature adaptive network further includes the following: A task-adaptive convolution is proposed, in which a common weight matrix, a classification weight matrix and a regression weight matrix are inserted in parallel next to the weight matrix of the convolutional neural network layer; the common weight represents a common feature subspace, which is used to learn common features between different tasks to ensure basic commonality between various tasks; the classification weight matrix represents a classification feature subspace, which is used to capture scale-invariant features and contextual information; the regression weight matrix represents a regression feature subspace, which is used to learn spatial detail information to facilitate regression calculation; Based on the task adaptive convolution, a feature adaptive network is proposed. The feature adaptive network concatenates the search area features and the response feature map as its input, then reduces the number of channels through a task adaptive convolution and then performs an average pooling operation on its spatial dimension; then, a task adaptive convolution and a linear rectification activation function are used to keep the number of channels unchanged; and then a task adaptive convolution and a logical activation function are used to learn and capture important channel information of different tasks, and finally a task adaptive convolution is used to obtain the final output.
4. A lightweight target tracking system based on hybrid linear attention according to claim 1, characterized in that: The input of the classification prediction head is the output of the feature adaptation network spliced together without including the regression feature subspace; the input of the regression prediction head is the output of the feature adaptation network spliced together without including the classification feature subspace.