Target tracking method, system, device and medium based on selective state space

Abstract

The present invention belongs to the field of image processing technology, and specifically relates to a target tracking method, system, device, and medium based on a selective state space. The method first introduces a proxy token to interact with features to be processed to capture contextual information, better focus on features related to the current frame target, and improve target recognition capabilities; then a selective state space is constructed to dynamically update the state of the current frame, effectively combine historical information with the current frame features, and dynamically adjust the expression of the current state; finally, the state of the current frame predicts a bounding box through a self-attention mechanism and linear transformation to perform target tracking, thereby improving the tracking accuracy and continuity of specific targets in a video sequence and enhancing the adaptability in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a target tracking method, system, device and medium based on selective state space. Background Art

[0002] Object tracking is a computer vision task that aims to detect and track specific objects in video sequences in real time. This technology is widely used in autonomous driving, video surveillance, human-computer interaction, and augmented reality. Through effective object tracking, the system can obtain the target's motion trajectory and behavior pattern, thereby making more intelligent decisions.

[0003] Although target tracking technology has made some progress in recent years, it still faces some bottlenecks in practical applications. Existing tracking algorithms often focus on optimizing backbone networks or feature processing, focusing on feature extraction within a single frame, while ignoring the dynamic changes of the target in the time series. This makes it difficult for the model to fully utilize temporal information, thus affecting the continuity and accuracy of tracking.

[0004] In addition, existing tracking methods usually adopt a divide-and-conquer strategy to decompose the tracking problem into multiple subtasks, such as target scale estimation and center point localization, each of which is addressed by a specific head network. This divide-and-conquer strategy has shown superior performance on tracking benchmarks, thus becoming the mainstream design in existing models. However, there are still two shortcomings. First, each subtask requires a customized head network, which leads to a complex tracking framework. Second, each head network requires one or more learning loss functions. Summary of the invention

[0005] The present invention provides a target tracking method, system, device and medium based on selective state space.

[0006] The technical solution of the present invention is as follows:

[0007] The present invention provides a target tracking method based on selective state space, comprising the following steps:

[0008] S1: Obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed;

[0009] The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token;

[0010] The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature.

[0011] The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature.

[0012] S2: Get the state of the previous frame of the current frame, combine the interactive features and the learnable matrix of the current frame, construct the selective state space and get the state of the current frame;

[0013] S3: The state of the current frame is combined with the projection matrix to generate the target features;

[0014] The target features are processed by the self-attention mechanism to obtain fusion features;

[0015] The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

[0016] In step S2, the state of the frame before the current frame is obtained, and the selective state space is constructed and the state of the current frame is obtained by combining the interactive features and the learnable matrix of the current frame, specifically:

[0017] According to the formula: , construct the selective state space and get the state of the current frame;

[0018] In the formula, is the state of the current frame; The state of the frame before the current frame; is the interactive feature; is the learnable matrix of the current frame; is the state transfer matrix.

[0019] In step S3, the state of the current frame is combined with the projection matrix to generate target features, specifically:

[0020] According to the formula: , generate target features;

[0021] In the formula, is the target feature; is the projection matrix; The status of the current frame.

[0022] In step S3, the fused features are linearly transformed to obtain bounding box coordinates, specifically:

[0023] According to the formula: , get the bounding box coordinates;

[0024] In the formula, are the bounding box coordinates; is the weight matrix; For fusion features; is the bias term.

[0025] In step S3, the target features are processed by the self-attention mechanism to obtain fusion features, specifically:

[0026] The target features are linearly transformed to generate a second query matrix, a second key matrix and a second value matrix. The second query matrix and the second key matrix are dot-producted to obtain attention weights. The attention weights are weighted summed on the second value matrix to obtain fused features.

[0027] In step S3, after obtaining the bounding box coordinates, non-maximum suppression processing is also performed on the bounding box coordinates.

[0028] The present invention also provides a target tracking system based on selective state space, comprising:

[0029] Feature processing module: used to obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed;

[0030] The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token;

[0031] The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature.

[0032] The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature.

[0033] Selective state space construction module: used to obtain the state of the previous frame of the current frame, combine the interactive features and the learnable matrix of the current frame, construct the selective state space and obtain the state of the current frame;

[0034] Target tracking module: used to combine the state of the current frame with the projection matrix to generate target features;

[0035] The target features are processed by the self-attention mechanism to obtain fusion features;

[0036] The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

[0037] The present invention also provides a target tracking device based on selective state space, comprising a processor and a memory, wherein the processor implements the target tracking method based on selective state space when executing a computer program stored in the memory.

[0038] The present invention also provides a target tracking medium based on selective state space, which is used to store a computer program, wherein the computer program implements the target tracking method based on selective state space when executed by a processor.

[0039] Beneficial effects: The present invention first introduces proxy tokens to interact with the features to be processed to capture contextual information, better focus on the features related to the current frame target, and improve the target recognition ability; then constructs a selective state space to dynamically update the state of the current frame, effectively combines historical information with the current frame features, and dynamically adjusts the expression of the current state; finally, the state of the current frame predicts the bounding box through the self-attention mechanism and linear transformation, and performs target tracking, which improves the tracking accuracy and continuity of specific targets in the video sequence and enhances the adaptability in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a target tracking method based on a selective state space is provided in an embodiment. DETAILED DESCRIPTION

[0041] The following examples are intended to illustrate the present invention rather than to further limit the present invention.

[0042] The present invention provides a target tracking method based on selective state space, such as Figure 1 As shown, the following steps are included:

[0043] S1: Obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed.

[0044] The template area refers to a specific image area in a video sequence that is used to represent the target object and represents the features of the stored known target. The search area refers to the area in the current frame that is used to find the area that best matches the template area, that is, to process potential targets in the current frame. As for the specific feature extraction operation in the target tracking task, the Vision Transformer (ViT) can be used as the backbone network to implement it.

[0045] In order to enhance the recognition ability of the target, it is necessary to interactively process the features to be processed so that the important features related to the target in the current frame can be focused on. The specific operations are as follows:

[0046] The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token;

[0047] The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature.

[0048] The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature.

[0049] The present invention captures context information by introducing proxy tokens, better focusing on features related to the current frame target, thereby improving the target recognition capability and achieving effective tracking.

[0050] S2: Get the state of the previous frame of the current frame, combine the interactive features and the learnable matrix of the current frame, construct the selective state space and get the state of the current frame, specifically:

[0051] According to the formula: , construct the selective state space and get the state of the current frame;

[0052] In the formula, is the state of the current frame; The state of the frame before the current frame; is the interactive feature; It is a learnable matrix of the current frame, which can be dynamically adjusted according to the current input, thus helping to selectively propagate important information related to the target while forgetting unnecessary features and enhancing the adaptability to time series data; is the state transfer matrix.

[0053] In order to better process time series data in target tracking, the present invention constructs a selective state space, dynamically updates the state of the current frame, effectively combines historical information with the current frame features, and dynamically adjusts the expression of the current state.

[0054] S3: The state of the current frame is combined with the projection matrix to generate the target features;

[0055] The target features are processed by the self-attention mechanism to obtain fusion features;

[0056] The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

[0057] Preferably, the state of the current frame is combined with the projection matrix to generate the target features, specifically:

[0058] According to the formula: , generate target features;

[0059] In the formula, is the target feature; is the projection matrix; The status of the current frame.

[0060] After getting the state of the current frame in step S2, we need to predict the target bounding box based on it, which can be achieved through 3SDecoder (selective state space decoder). First, the state of the current frame is processed to generate target features. This operation can make full use of the current state and context features to generate target predictions more accurately.

[0061] In addition, the target features are processed by the self-attention mechanism to obtain fusion features, specifically:

[0062] The target features are linearly transformed to generate a second query matrix, a second key matrix and a second value matrix. The second query matrix and the second key matrix are dot-producted to obtain attention weights. The attention weights are weighted summed on the second value matrix to obtain fused features.

[0063] After being processed by the self-attention mechanism, the fusion feature contains the current state The fusion features of the important context information avoid the direct prediction of the interactive features, reduce the complex prediction process, and lay the foundation for the subsequent bounding box prediction.

[0064] In addition, the fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

[0065] Specifically:

[0066] According to the formula: , get the bounding box coordinates;

[0067] In the formula, are the bounding box coordinates; is the weight matrix; For fusion features; is the bias term.

[0068] After linear transformation, the fused features are converted into bounding box coordinates, and the expression ability can be further enhanced through nonlinear activation functions.

[0069] Considering that redundant boxes may appear, preferably, after obtaining the bounding box coordinates, non-maximum suppression processing is also performed on the bounding box coordinates to eliminate redundant boxes and improve the accuracy and robustness of the prediction, ensuring that the obtained bounding box prediction effect is the best. It can be expressed as:

[0070] ;

[0071] In the formula, are the bounding box coordinates; Represents the bounding box after Non-Maximum Suppression (NMS) processing.

[0072] The present invention first introduces proxy tokens to interact with the features to be processed to capture contextual information, better focus on the features related to the current frame target, and improve the target recognition ability. Then, a selective state space is constructed to dynamically update the state of the current frame, effectively combining historical information with the current frame features and dynamically adjusting the expression of the current state. Finally, the state of the current frame is used to predict the bounding box through a self-attention mechanism and linear transformation to perform target tracking, thereby improving the tracking accuracy and continuity of specific targets in video sequences and enhancing the adaptability in complex scenes.

[0073] The present invention also provides a target tracking system based on selective state space, comprising:

[0074] Feature processing module: used to obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed;

[0075] The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token;

[0076] The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature.

[0077] The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature.

[0078] Selective state space construction module: used to obtain the state of the previous frame of the current frame, combine the interactive features and the learnable matrix of the current frame, construct the selective state space and obtain the state of the current frame;

[0079] Target tracking module: used to combine the state of the current frame with the projection matrix to generate target features;

[0080] The target features are processed by the self-attention mechanism to obtain fusion features;

[0081] The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

[0082] The present invention also provides a target tracking device based on selective state space, comprising a processor and a memory, wherein the processor implements the target tracking method based on selective state space when executing a computer program stored in the memory.

[0083] The present invention also provides a target tracking medium based on selective state space, which is used to store a computer program, wherein the computer program implements the target tracking method based on selective state space when executed by a processor.

Claims

1. A target tracking method based on selective state space, characterized in that: The following steps are involved: S1: Obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed; The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token; The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature. The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature. S2: Get the state of the previous frame of the current frame, combine the interactive features and the learnable matrix of the current frame, according to the formula: , construct the selective state space and get the state of the current frame; where, is the state of the current frame; The state of the frame before the current frame; is the interactive feature; is the learnable matrix of the current frame; is the state transfer matrix; S3: According to the formula: , the state of the current frame is combined with the projection matrix to generate the target feature; Formula middle, is the target feature; is the projection matrix; is the state of the current frame; The target features are processed by the self-attention mechanism to obtain fusion features; The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

2. The target tracking method based on selective state space according to claim 1, characterized in that: In step S3, the fused features are linearly transformed to obtain bounding box coordinates, specifically: According to the formula: , get the bounding box coordinates; In the formula, are the bounding box coordinates; is the weight matrix; For fusion features; is the bias term.

3. The target tracking method based on selective state space according to claim 1, characterized in that: In step S3, the target features are processed by the self-attention mechanism to obtain fusion features, specifically: The target features are linearly transformed to generate a second query matrix, a second key matrix and a second value matrix. The second query matrix and the second key matrix are dot-producted to obtain attention weights. The attention weights are weighted summed on the second value matrix to obtain fused features.

4. The target tracking method based on selective state space according to claim 1, characterized in that: In step S3, after obtaining the bounding box coordinates, non-maximum suppression processing is also performed on the bounding box coordinates.

5. A target tracking system based on selective state space, characterized in that: include: Feature processing module: used to obtain the template area image and the search area image, extract features respectively, and then splice them to obtain the features to be processed; The features to be processed are linearly transformed to generate a first query matrix, a first key matrix and a first value matrix, and the first query matrix is ​​pooled to generate a proxy token; The proxy attention weight is calculated by using the dot product of the proxy token and the first key matrix and the position bias. The proxy attention weight is weighted on the first value matrix to obtain the proxy feature. The interaction feature attention weight is calculated by using the dot product of the first query matrix and the proxy token and the proxy bias. The interaction feature attention weight is dot-producted with the proxy feature to obtain the interaction feature. Selective state space building module: used to obtain the state of the previous frame of the current frame, combining the interactive features and the learnable matrix of the current frame, according to the formula: , construct the selective state space and get the state of the current frame; where, is the state of the current frame; The state of the frame before the current frame; is the interactive feature; is the learnable matrix of the current frame; is the state transfer matrix; Target tracking module: used according to the formula: , the state of the current frame is combined with the projection matrix to generate the target feature; Formula middle, is the target feature; is the projection matrix; is the state of the current frame; The target features are processed by the self-attention mechanism to obtain fusion features; The fused features are linearly transformed to obtain the bounding box coordinates for target tracking.

6. A target tracking device based on selective state space, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the target tracking method based on the selective state space as described in any one of claims 1 to 4 is implemented.

7. A target tracking medium based on a selective state space, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the target tracking method based on the selective state space as described in any one of claims 1 to 4 is implemented.

Images