Visual target tracking method based on K-est attention mechanism and LG convolution
By combining K-est attention mechanism and LG convolution in visual target tracking, feature relationships are captured from global and local perspectives, the problem of poor tracking performance in complex scenarios is solved, and the target tracking effect with high precision and high real-time performance is achieved.
Patent Information
- Application Number
- CN202410854356.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2025-02-25
AI Technical Summary
The traditional Siamese-based tracking method ignores local feature relationships when capturing global dependencies of features, resulting in reduced discernibility between the target and the background, and poor tracking performance in complex scenarios; while the Transformer-based tracker can capture feature dependencies from a global perspective, it ignores local feature relationships, resulting in insufficient tracking accuracy.
The visual target tracking method based on K-est attention mechanism and LG convolution are used to capture feature relationships from global and local perspectives. Only the most relevant dependencies of the K elements are considered through the K-est attention mechanism, and local feature relationships are captured from 9 directions in combination with LG convolution, and feature fusion is performed to improve model representation learning ability.
It effectively improves the accuracy and performance of target tracking, allowing the tracker to capture targets more accurately in complex scenarios, avoiding waste of computing resources caused by frequent updates, and achieving a balance between high real-time and high precision.
Smart Images
Figure CN119515913A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of visual target tracking, and in particular relates to a visual target tracking method based on K-est attention mechanism and LG convolution. Background Art
[0002] Object tracking has always been an important and challenging topic in the field of computer vision. Its task is to give the real target box of the target in the first frame of the video sequence, which contains the position and size information of the tracked target. The algorithm uses the real target information of the first frame to predict the position and size of the target in the next frame. The returned position information is usually represented by the coordinates of the rectangular bounding box that tightly surrounds the target. In recent years, with the development of deep learning, object tracking has been widely used in real-world scenarios, including video surveillance, autonomous driving, drone navigation, virtual reality, augmented reality, and smart security.
[0003] However, in order to cope with the problem of target tracking in complex scenes, the traditional Siamese-based tracking method only uses convolution operations to model feature relationships, which limits the receptive field to the convolution kernel and is not good at capturing the global feature dependencies. The current mainstream Transformer-based tracker uses the attention mechanism to capture feature dependencies from a global perspective while ignoring local feature relationships, which reduces the discriminability of the target and the background, limits the tracking performance in complex scenes, and leads to resource waste and insufficient tracking accuracy. Summary of the invention
[0004] The purpose of the present invention is to provide a visual target tracking method based on K-est attention mechanism and LG convolution, which can capture feature relationships from global and local perspectives, avoid the waste of computing resources caused by frequent updates, and improve the tracking effect of the tracker.
[0005] The technical solution to achieve the purpose of the present invention is:
[0006] A visual target tracking method based on K-est attention mechanism and LG convolution, including:
[0007] Step 1: extract features from the search area and template through a backbone feature extraction network to obtain template features and search area features;
[0008] Step 2: The search area features are processed and encoded through the K-est attention mechanism and the LG convolution module to obtain the encoded search area features, and the template features are processed and encoded through the multi-head self-attention module to obtain the encoded template features. Finally, the encoded search area features and the encoded template features are fused through the fusion module to obtain the fused features.
[0009] Step 3: The target positioning module extracts fusion features through the prediction head and calculates the mathematical expectation value to predict the corner points to obtain the prediction box coordinates;
[0010] Step 4: Perform iterative training of steps 1 to 3 through a two-stage loss function, update the parameters of the backbone feature extraction network, K-est attention mechanism and LG convolution module, multi-head self-attention module, fusion module and target positioning module, and dynamically update the template during iterative training to obtain a network model that meets the requirements for target tracking.
[0011] Furthermore, the backbone feature extraction network in step 1 adopts a ResNet50 network with the three residual blocks in the final stage and the last fully connected layer removed.
[0012] Furthermore, the K-est attention mechanism in step 2 only considers the dependencies of the most relevant K elements. First, the attention score is obtained by matrix multiplication between the query vector Query and the key vector Key. Then, after attention scaling and attention masking operations, the attention score is input into the K-est selection module. In the K-est selection module, the first K attention scores of each row are retained, while the rest are set to -∞. After the softmax operation, the first K K-est attention scores generate K-est attention weights, in which the weights of the first K elements are increased, while the weights of other elements are equal to 0. Finally, the K-est attention weights are multiplied with the value vector Value to obtain the final K-est attention result.
[0013] Furthermore, the LG convolution includes: the input feature is divided into N g Each group is convolved with a learnable group convolution kernel to capture local feature relationships from 9 directions. Finally, the convolution features are concatenated together along the spatial dimension to obtain the final convolution features.
[0014] Furthermore, the fusion module includes a multi-head cross-attention module, a residual connection layer, a normalization layer, and a feedforward network. The fusion module is used to fuse the encoding search area features and the encoding template features to obtain fused features, specifically including: the encoding search area features and the encoding template features are fused through the multi-head cross-attention module, after the residual connection layer and the normalization layer are operated, the fused features are generated, and then the feedforward network is used to generate the final fused features.
[0015] Furthermore, the final fusion feature is:
[0016]
[0017] in, Respectively represent the encoding template features, Xf Respectively represent the encoding search area features, intermediate fusion features and final fusion features, Pos z and Pos x They represent the spatial position encoding of the template and the search area respectively, LN represents the normalization layer, MHCA represents the multi-head cross attention, and FFN represents the feed network.
[0018] Furthermore, the predicted frame coordinates obtained by the target positioning module in step 3 are:
[0019]
[0020] Among them, P tl (x,y) and P br (x, y) are the probabilities of the upper left corner and lower right corner of the bounding box, respectively, which are obtained by inputting the fusion features extracted by the prediction head into the FCN network prediction, that is, P tl (z,y),P br (x,y)=FCN(X f ), and To calculate the predicted box coordinates expected by the probability distribution of corner points.
[0021] Furthermore, the first stage loss function is:
[0022]
[0023] Among them, λ L1 , iou represents the weights of the two loss functions, b i and Refers to the real bounding box and predicted bounding box of the tracked target, L1, L iou represents L1 loss and IoU loss.
[0024] Furthermore, the second stage loss function is:
[0025] L score =y i log(p i )+(1-y i )log(1-p i )
[0026] Among them, p i represents the prediction score, y i Represents the actual label value.
[0027] Furthermore, a template dynamic update mechanism that is dual-triggered by time interval and classification score is used to dynamically update the template.
[0028] Compared with the existing target tracking model, the advantages and beneficial effects of the present invention are as follows: the present invention adopts a backbone feature extraction network designed by removing the three residual blocks and the last fully connected layer of the last stage of ResNet50, which is universal and can be easily integrated into various convolutional network structures; combined with the K-est attention module and the LG convolution structure, the model representation learning ability is improved, so that it can capture feature relationships from a global and local perspective; the target positioning module that calculates the mathematical expectation value to predict the corner point can improve the accuracy of the target positioning frame prediction; the template dynamic update mechanism that is dual-triggered by the time interval and the classification score can timely update the template to adapt to the target change when the target appearance changes significantly, and can avoid the waste of computing resources caused by frequent updates, thereby achieving a balance between high real-time performance and high precision; the two-stage loss function of L1 loss and IoU loss and cross entropy loss is adopted to further improve the border regression effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 The figure is an overall flow chart of the method of the present invention.
[0030] Figure 2 A diagram representing the learning method in the present invention.
[0031] Figure 3 This is a structural diagram of the AC module in the present invention.
[0032] Figure 4 This is the structure diagram of the K-est attention mechanism in the present invention.
[0033] Figure 5 This is the LG convolution structure diagram in the present invention. DETAILED DESCRIPTION
[0034] The present invention will be further described below in conjunction with the accompanying drawings.
[0035] A visual target tracking method based on K-est attention mechanism and LG convolution, combined with Figure 1 , the method comprises the following steps:
[0036] 1) For the search area and template, feature extraction is performed using the ResNet50 backbone feature extraction network without the three residual blocks in the last stage and the last fully connected layer.
[0037] 2) The extracted features are fused using a feature fusion module that includes a K-est attention mechanism and LG convolution. The K-est attention is used to capture the long-range feature dependencies in the target tracking task and focus on the primary target information. The lightweight LG convolution is used to provide additional inductive bias to highlight the discrimination boundary between the target and the background.
[0038] 3) Based on the fused features, the method of calculating the mathematical expectation value is used to predict and locate the target corner points.
[0039] 4) Adopting the template dynamic update mechanism triggered by time interval and classification score to update the tracking template; using the intermediate frame to obtain the dynamic template, capturing the change of target appearance over time and providing additional timing information, and updating the dynamic template frame when a certain time interval is reached and the classification score reaches a certain threshold;
[0040] 5) A two-stage loss function combining L1 loss and IoU loss with cross entropy loss is used for model training to obtain a network model that meets the requirements; in order to prevent the main body positioning ability of the tracker from being affected, all parameters except the template score head parameters are frozen.
[0041] The backbone feature network uses the ResNet50 structure with the three residual blocks and the last fully connected layer of the last stage removed for feature extraction, and the template image and the search area image are input into it for feature extraction to obtain the template features and the search area features.
[0042] Combination Figure 2 ,The step 2 combines the attention mechanism and the convolution operation to capture ,feature relationships from both global and local perspectives, where the K-est attention ,features and LG convolution features are fused in a certain ratio.
[0043] LG convolution and K-est attention mechanism. This structure contains a feature fusion module designed with K-est attention mechanism and LG convolution structure. Template features and search area features are input into this structure to generate fusion features. This module consists of a template branch and a search area branch. Figure 2 The AC module proposed in the present invention. This module consists of a template branch and a search area branch. For the template branch, after the template feature is fused with the spatial position encoding, it is input into the multi-head self-attention (MHSA) module for feature encoding, and then after the residual connection and layer normalization operations, the encoding features are generated. For the search area branch, K-est attention and LG convolution are used to carry out global and local feature representation learning. After the search area features are processed by K-est attention and LG convolution, encoding features rich in semantic information are generated. Then, the encoded feature vector is sent to the multi-head cross attention (MHCA) module for feature fusion. After the residual connection and layer normalization (LN) operations, the fused features are generated. In order to enhance the linear fitting ability of the features, the method of the present invention uses a feedforward network (FFN) to generate the final fused features. The formula is used:
[0044]
[0045] Among them, Z, Represent template features and encoded template features respectively. X, X f Respectively represent the search area feature, the encoded search area feature, the intermediate fusion feature and the final fusion feature. z and Pos x Represents the spatial position encoding of the template and the search area respectively. Kest , X LG They refer to K-est attention features and LG convolution features respectively.
[0046] In addition, K-est attention features and LG convolution features are fused in a certain ratio. The feature fusion formula is as follows:
[0047] ACMix(X Kest ,X LG )=ω α ·X Kest +ω c ·X LG (2)
[0048] Among them, ω α and ω c Represents the fusion weight of attention and convolution features. ,
[0049] Figure 3 This is the K-set attention mechanism structure diagram, combined with Figure 3 , the K-est attention mechanism only considers the dependencies between an element and its most relevant K elements. Given a Query, Key, and Value, the attention score is first obtained by matrix multiplication between the Query and Key. Then, after the attention scaling scale and attention mask operations, the attention score is input into the K-est selection module. In the K-est selection module, only the top K attention scores of each row are retained, while the rest are set to -∞. After the softmax operation, the K-est attention score generates the K-est attention weight, in which the weights of the top K elements are significantly increased, while the weights of other elements are equal to 0. Finally, the K-est attention weight is multiplied by the Value to obtain the final K-est attention result, using the formula:
[0050]
[0051] Among them, a ij and w ij denote the attention score and attention weight of the i-th row and j-th column, respectively, and denotes the k-th maximum value of the i-th column.
[0052] The computational complexity of traditional convolution is too high. This method uses lightweight LG convolution to capture local feature dependencies. Its translation invariance and local correlation make it better at capturing local feature dependencies than the attention mechanism, which helps to distinguish foreground and background. The input features are divided into N g Each group is convolved with a learnable group convolution kernel to capture local feature relationships from 9 directions. Finally, the convolution features are concatenated along the spatial dimension to obtain the final convolution features. The computational complexity of LG convolution is only 1 / 2 of that of traditional convolution.
[0053] The target positioning module adopts the method of calculating the mathematical expectation value to predict the corner points. Different from the previous anchor box-based method, the prediction head of this method predicts the target position based on the corner point. Specifically, an FCN network is used to predict the probability of the target corner point at each position, and then the overall mathematical expectation of the corner point is calculated to obtain the position of the upper left corner and the lower right corner of the final target bounding box to achieve target positioning.
[0054]
[0055]
[0056] in:
[0057] P tl (x,y),P br (x,y)=FCN(X f ) (7)
[0058] P tl (x,y) and P br (x,y) are the probabilities of the upper left and lower right corners of the bounding box, respectively. and To calculate the predicted box coordinates expected by the probability distribution of corner points.
[0059] The dynamic template update mechanism is triggered by both time interval and classification score. As the traditional fixed template cannot effectively deal with the problem that the target appearance changes significantly over time, the intermediate frame information is used to generate and update the template. First, the position and appearance information of the target in the first frame are determined through the initialization process to generate the basic template. As time goes by, when the set time interval T is reached, the template is updated. u , the system will analyze the intermediate frames containing potential target areas in the continuous video frames. When the classification score of an intermediate frame exceeds the preset threshold τ, it indicates that the feature expression of the target object in the frame is highly reliable and can accurately reflect the current state of the target. At this time, the system automatically uses the feature representation of this intermediate frame as a new dynamic template to replace the original static template to capture the appearance changes and motion characteristics of the target over time.
[0060] When calculating the loss function, a two-stage loss function of L1 loss, IoU loss and cross entropy loss is used. In the first stage, the main body of the tracker is trained, and its loss function consists of L1 loss and IoU loss. The calculation formula is as follows:
[0061]
[0062] Among them, λ L1 , iou represents the weights of the two loss functions, b i and Refers to the true bounding box and predicted bounding box of the tracked object.
[0063] In the second stage, the cross entropy loss is used to train the score head. In order to prevent the tracker's subject localization ability from being affected, all parameters except the template score head parameters are frozen. The score head training loss is defined as follows:
[0064] L score =y i log(p i )+(1-y i )log(1-p i ) (9)
[0065] Among them, p i represents the prediction score, y i Represents the actual label value.
[0066] The method proposed in the present invention combines the attention mechanism and the convolution operation, which can capture feature relationships from both global and local perspectives, making the tracking performance significantly better than the previous tracker that only uses the attention mechanism, and the tracking speed is not reduced. As a general visual target tracker, the method of the present invention can be applied to many computer vision target tracking fields, such as video surveillance, unmanned driving, mobile robots, drone navigation, etc.
Claims
1. A visual target tracking method based on K-est attention mechanism and LG convolution, characterized in that: include: Step 1: extract features from the search area and template through a backbone feature extraction network to obtain template features and search area features; Step 2: The search area features are processed and encoded through the K-est attention mechanism and the LG convolution module to obtain the encoded search area features, and the template features are processed and encoded through the multi-head self-attention module to obtain the encoded template features. Finally, the encoded search area features and the encoded template features are fused through the fusion module to obtain the fused features. Step 3: The target positioning module extracts fusion features through the prediction head and calculates the mathematical expectation value to predict the corner points to obtain the prediction box coordinates; Step 4: Perform iterative training of steps 1 to 3 through a two-stage loss function, update the parameters of the backbone feature extraction network, K-est attention mechanism and LG convolution module, multi-head self-attention module, fusion module and target positioning module, and dynamically update the template during iterative training to obtain a network model that meets the requirements for target tracking.
2. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The backbone feature extraction network in step 1 adopts the ResNet50 network with the three residual blocks in the last stage and the last fully connected layer removed.
3. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The K-est attention mechanism in step 2 only considers the dependencies of the most relevant K elements. First, the attention score is obtained by matrix multiplication between the query vector Query and the key vector Key. Then, after attention scaling and attention masking operations, the attention score is input into the K-est selection module. In the K-est selection module, the first K attention scores of each row are retained, while the rest are set to -∞. After the softmax operation, the first K K-est attention scores generate K-est attention weights, in which the weights of the first K elements are increased, while the weights of other elements are equal to 0. Finally, the K-est attention weights are multiplied with the value vector Value to obtain the final K-est attention result.
4. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The LG convolution includes: the input feature is divided into N g Each group is convolved with a learnable group convolution kernel to capture local feature relationships from 9 directions. Finally, the convolution features are concatenated together along the spatial dimension to obtain the final convolution features.
5. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The fusion module includes a multi-head cross attention module, a residual connection layer, a normalization layer, and a feedforward network; The coding search area feature and the coding template feature are fused through a fusion module to obtain a fused feature, which specifically includes: the coding search area feature and the coding template feature are fused through a multi-head cross attention module, after the residual connection layer and the normalization layer are operated, the fused feature is generated, and then the feedforward network is used to generate the final fused feature.
6. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 5, characterized in that: The final fusion features are: in, Respectively represent the encoding template features, X f Respectively represent the encoding search area features, intermediate fusion features and final fusion features, Pos z and Pos x They represent the spatial position encoding of the template and the search area respectively, LN represents the normalization layer, MHCA represents the multi-head cross attention, and FFN represents the feed network.
7. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The predicted frame coordinates obtained by the target positioning module in step 3 are: Among them, P tl (x,y) and P br (x, y) are the probabilities of the upper left corner and lower right corner of the bounding box, respectively, which are obtained by inputting the fusion features extracted by the prediction head into the FCN network prediction, that is, P tl (x,y),P br (x,y)=FCN(X f ), and To calculate the predicted box coordinates expected by the probability distribution of corner points.
8. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The loss function of the first stage is: Among them, λ L1 , iou represents the weights of the two loss functions, b i and Refers to the real bounding box and predicted bounding box of the tracked target, L1, L iou represents L1 loss and IoU loss.
9. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The loss function of the second stage is: L score =y i log(p i )+(1-y i )log(1-p i ) Among them, p i represents the prediction score, y i Represents the actual label value.
10. The visual target tracking method based on K-est attention mechanism and LG convolution according to claim 1, characterized in that: The template is dynamically updated by using a template dynamic update mechanism that is dual-triggered by time intervals and classification scores.