Target tracking method and system based on Mamba visual hybrid module
By combining the Mamba visual hybrid module with Transformer and Mamba state space model, the problem of insufficient global information capture in the prior art is solved, and a more accurate and robust target tracking effect is achieved.
Patent Information
- Application Number
- CN202510777185.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-11
- Publication Date
- 2025-08-29
- Estimated Expiration
- 2045-06-11
AI Technical Summary
The existing Mamba-based target tracking technology is difficult to capture the global information of the image when handling long-term tracking and facing occlusion, similar objects interference, resulting in tracking failure.
The Mamba visual hybrid module based on the Transformer architecture and the Mamba state space model is adopted, combining the multi-head self-attention and image state space modules to extract and superimpose global context information and local detail information, and feature enhancement is performed through feature fusion and feature improvement modules to optimize the tracking model.
It improves the accuracy and robustness of target tracking, can maintain efficient tracking performance in complex scenarios, enhances the capture of local semantic information, and avoids the loss of global information.
Smart Images
Figure CN120298458B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a target tracking method and system based on a Mamba vision hybrid module. Background Art
[0002] Object tracking is a fundamental task in computer vision applications. Given a target, it aims to locate and label a target object in consecutive video frames. It is currently widely used in fields such as autonomous driving, biomedicine, indoor and outdoor surveillance, highway intersection videography, drones and robotics, and human-computer interaction. Despite rapid progress in recent years, practical applications still face challenges such as occlusion, deformation, background blur, and interference from similar objects. Therefore, there is still significant room for improvement in object tracking technology.
[0003] The emergence and application of the Transformer have further advanced object tracking technology. It uses an attention mechanism to capture contextual information from video images. By reducing losses through a multi-layer network architecture and backpropagation algorithms, it automatically focuses on areas of interest, thereby improving tracking performance. Currently, most tracking algorithms are based on the Transformer and have achieved excellent tracking results. However, the attention mechanism focuses on capturing global information, losing much critical local information, which can significantly impact tracker performance. Combining local and global information is a key approach to improving tracker performance.
[0004] The Mamba architecture, based on the state-space model, is widely used in various fields. The state-space model efficiently processes data through selective scanning, and the Mamba architecture based on this reduces computational complexity and enables low-latency inference. With far fewer model parameters than the Transformer, some applications can achieve the same results. The latest Mamba-based trackers have significantly improved tracking speed while maintaining tracking performance. However, their sequential data processing method has spatial dependency limitations, making it difficult to capture the global information of the image. This often leads to tracking failures during long-term tracking and in the face of occlusion, interference from similar objects, and sudden disappearances. Summary of the Invention
[0005] In view of the above situation, the main purpose of the present invention is to propose a target tracking method and system based on the Mamba visual hybrid module to solve the above technical problems.
[0006] The present invention proposes a target tracking method based on the Mamba visual hybrid module, the method comprising the following steps:
[0007] Step 1: Construct a Mamba visual hybrid module based on the Transformer architecture and the Mamba state space model, a feature fusion module based on the feature aggregation mechanism, and a feature enhancement module based on the feature enhancement mechanism. The Mamba visual hybrid module, the feature fusion module, and the feature enhancement module constitute the tracking model; the Mamba visual hybrid module includes a multi-head self-attention and image state space module;
[0008] Step 2: Input the search image and template image into the Mamba visual mixing module, extract and superimpose global context information and local detail information, and obtain the output of the Mamba visual mixing module;
[0009] Step 3: Input the output of the Mamba visual mixing module into the feature fusion module for fusion to obtain the output of the feature fusion module;
[0010] Step 4: The output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The enhanced features are used to calculate the classification loss and regression loss respectively. The classification loss and regression loss are used to optimize the tracking model to obtain the optimized tracking model.
[0011] Step 5: Pre-train the optimized tracking model using a large-scale data set, and update the parameters of the tracking model to obtain a tracking model with updated parameters;
[0012] Step 6: Input the search image and template image into the tracking model with updated parameters, repeat steps 2 to 4 in an iterative manner to obtain the final enhanced features, and input the final enhanced features into the prediction head to obtain the tracking results.
[0013] The present invention also proposes a target tracking system based on the Mamba visual hybrid module, the system comprising:
[0014] Building blocks for:
[0015] The Mamba visual hybrid module is built based on the Transformer architecture and the Mamba state space model. The feature fusion module is built based on the feature aggregation mechanism. The feature enhancement module is built based on the feature enhancement mechanism. The Mamba visual hybrid module, feature fusion module, and feature enhancement module constitute the tracking model. The Mamba visual hybrid module includes multi-head self-attention and image state space modules.
[0016] Extraction module for:
[0017] The search image and template image are input into the Mamba visual mixing module, global context information and local detail information are extracted and superimposed, and the output of the Mamba visual mixing module is obtained;
[0018] Compute module for:
[0019] The output of the Mamba visual mixing module is input into the feature fusion module for fusion to obtain the output of the feature fusion module;
[0020] Learning modules for:
[0021] The output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The enhanced features are used to calculate the classification loss and regression loss respectively. The classification loss and regression loss are used to optimize the tracking model to obtain the optimized tracking model.
[0022] Pre-training modules for:
[0023] Pre-training the optimized tracking model using a large-scale dataset and updating the parameters of the tracking model to obtain a tracking model with updated parameters;
[0024] Tracking module for:
[0025] The search image and template image are input into the tracking model with updated parameters. The global context information and local detail information are extracted and superimposed in an iterative manner using the Mamba visual mixing module, and fused through the feature fusion module. Finally, the feature enhancement module is used to enhance the features to obtain the final enhanced features. The final enhanced features are input into the prediction head to obtain the tracking results.
[0026] Compared with the prior art, the present invention has the following beneficial effects:
[0027] 1. The Mamba visual blending module designed in this paper adopts a dual-branch architecture of attention and image state. The attention branch captures long-range image dependencies by calculating multi-head attention to obtain global contextual information. The image state branch not only dynamically selects features but also preserves spatial details through a symmetric convolutional structure. This is parameter-efficient and can capture local semantic information. The superposition of the two enhances local semantic information while avoiding the loss of global information.
[0028] 2. The present invention designs a feature aggregation network, uses the image state space module to integrate the historical state information of the search area, splices the historical state information with the interacted features along the channel direction, and realizes effective feature aggregation through convolution, activation, normalization, and jump connection to filter out irrelevant information.
[0029] 3. This invention designs a feature enhancement module, which consists of two parts, connected in series. The first part performs multi-layer channel attention calculations on the input and then aggregates multiple features. The second part follows the same structure and performs spatial attention calculations. This module models channel and spatial dependencies, simultaneously capturing global and local features, and enhancing the corresponding regional features of the target object.
[0030] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 This is a flowchart of the steps of the target tracking method based on the Mamba visual hybrid module proposed in the present invention.
[0032] Figure 2 This is a schematic diagram of the Mamba visual mixing module of the target tracking method based on the Mamba visual mixing module proposed in the present invention.
[0033] Figure 3 This is a schematic diagram of the feature enhancement module PFM of the target tracking method based on the Mamba visual hybrid module proposed in the present invention.
[0034] Figure 4 This is a tracking architecture diagram of the target tracking method based on the Mamba visual hybrid module proposed in this invention.
[0035] Figure 5 This is a structural diagram of the target tracking system based on the Mamba visual hybrid module proposed in this invention. DETAILED DESCRIPTION
[0036] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.
[0037] These and other aspects of the embodiments of the present invention will become clear with reference to the following description and accompanying drawings. In these descriptions and accompanying drawings, some specific implementations of the embodiments of the present invention are specifically disclosed to provide some ways to implement the principles of the embodiments of the present invention, but it should be understood that the scope of the embodiments of the present invention is not limited thereto.
[0038] See also Figure 1 The embodiment of the present invention proposes a target tracking method based on the Mamba visual hybrid module, which includes the following steps:
[0039] Step 1. Construct a Mamba visual mixing module based on the Transformer architecture and the Mamba state space model, construct a feature fusion module based on the feature aggregation mechanism, and construct a feature enhancement module based on the feature enhancement mechanism. The Mamba visual mixing module, feature fusion module, and feature enhancement module constitute the tracking model; the Mamba visual mixing module includes multi-head self-attention and image state space modules.
[0040] Step 2: Input the search image and template image into the Mamba visual mixing module, extract and superimpose global context information and local detail information, and obtain the output of the Mamba visual mixing module.
[0041] See also Figure 2 and Figure 4 In step 2, the search image and the template image are input into the Mamba visual mixing module to extract and superimpose the global context information and local detail information to obtain the output of the Mamba visual mixing module. The specific steps include the following:
[0042] Input the search image and template image into the Mamba visual mixing module for initialization processing respectively, and obtain the image sequence of the template area and the image sequence of the search area;
[0043] The image sequence of the template area and the image sequence of the search area are respectively subjected to feature extraction using a convolutional network to obtain the original features of the image sequence of the template area and the original features of the image sequence of the search area;
[0044] The original features of the template region image sequence are flattened and position-encoded in sequence to obtain the original feature label sequence of the template image;
[0045] The original features of the search area image sequence are flattened and position-coded in sequence to obtain the original feature label sequence of the search area image;
[0046] The original feature tag sequence of the template image is input into the multi-head self-attention to extract information and obtain the global context features of the image;
[0047] The original feature label sequence of the search area image is input into the image state space module to extract local information and obtain the local features of the image;
[0048] The global context features of the image are superimposed with the local features of the image to obtain the output of the Mamba visual mixing module.
[0049] The original feature tag sequence of the template image is input into the multi-head self-attention for information extraction. The corresponding process has the following relationship:
[0050] ;
[0051] in, represents the middle value, represents attention calculation, Represents the input of the Mamba visual mixing module, represents the normalization function, represents the query vector, represents the transpose of the key vector, represents the square root of the key vector dimension, represents a value vector;
[0052] In the step of inputting the original feature label sequence of the search area image into the image state space module to extract local information and obtain the local features of the image, the corresponding process has the following relationship:
[0053] ;
[0054] in, Representing a state-space model Current moment Input, represents the first learnable matrix, Representing a state-space model Past Moment The information set of all input images, represents the second learnable matrix, represents the third learnable matrix, represents the fourth learnable matrix, represents the matrix multiplication operation, represents the state branch of the image state space module, represents the symmetric convolution branch of the image state space module, Indicates the first linear mapping process, represents the output of the state branch of the image state space module, Indicates activation operation. represents a one-dimensional ordinary convolution operation, Indicates the second linear mapping process, Indicates a connection operation;
[0055] In the step of superimposing the global context features of the image with the local features of the image to obtain the output of the Mamba visual mixing module, the corresponding process has the following relationship:
[0056] ;
[0057] in, Represents the output of the Mamba visual mixing module.
[0058] Furthermore, the first frame template image and the search area images of the subsequent frames are initialized to obtain the image sequences of the template and the search area; the original features of the template and the search area image sequences are respectively extracted through the convolutional network, and the original features are flattened; then, by adding position encoding, the original feature labeling sequence of the template image and the original feature labeling sequence of the search area image are obtained.
[0059] It should be noted that the Transformer-based network architecture uses the Mamba visual hybrid module at the lower level to extract features from the template area and the search area. Figure 2 As shown in the figure, the Mamba visual mixing module consists of an image state space module and multi-head self-attention, which extracts global context information and local detail information of the image through a dual-branch structure; the first branch of the image state space module integrates image history information to avoid missing and incomplete information of the target object, and the second branch provides global information to enhance modeling capabilities. The two branches are combined to dynamically select key features and obtain local information of the image.
[0060] Step 3: Input the output of the Mamba visual mixing module into the feature fusion module for fusion to obtain the output of the feature fusion module.
[0061] In step 3, the output of the Mamba visual mixing module is input into the feature fusion module for fusion to obtain the output of the feature fusion module. The specific steps include the following:
[0062] The image state space module is used to integrate the historical state information of the search area to obtain the integrated historical state information of the search area;
[0063] The output of the Mamba visual mixing module is input into the feature fusion module, and the splicing operation, convolution activation operation and residual connection operation are performed in sequence with the historical state information of the integrated search area to obtain the output of the feature fusion module.
[0064] The output of the Mamba visual mixing module is input into the feature fusion module, and the historical state information of the integrated search area is sequentially subjected to splicing operations, convolution activation operations, and residual connection operations to obtain the output of the feature fusion module. The corresponding process has the following relationship:
[0065] ;
[0066] in, Represents the features output after concatenation and linear mapping, represents the fully connected layer, Represents the features after the interaction between the template area and the search area, represents the features output by the image state space module, Indicates fusion processing, represents the batch normalization operation, Represents a convolution operation with a convolution kernel size of 1×1, represents the activation function, Represents the output of the feature fusion module.
[0067] Step 4: Use the feature enhancement module to enhance the output of the feature fusion module to obtain enhanced features. Use the enhanced features to calculate the classification loss and regression loss respectively. Use the classification loss and regression loss to optimize the tracking model to obtain the optimized tracking model.
[0068] See also Figure 3 and Figure 4 In step 4, the output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The specific steps are as follows:
[0069] S101, performing global average pooling processing on the spatial dimension of the output of the feature fusion module using the feature lifting module to obtain a result after the global average pooling processing, performing channel allocation weight processing on the result after the global average pooling processing through a convolutional layer to obtain a channel weight matrix of the input feature, and multiplying the channel weight matrix of the input feature with the original matrix output by the feature fusion module to obtain a feature matrix of the first layer channel branch output and a matrix of the first layer skip connection output;
[0070] S102, using the matrix output by the first layer of skip connections as input and iteratively repeating step S101 to obtain the feature matrix output by the second layer of channel branches, the feature matrix output by the third layer of channel branches, and the feature matrix output by the fourth layer of channel branches respectively;
[0071] S103, concatenating the feature matrix output by the first layer channel branch, the feature matrix output by the second layer channel branch, the feature matrix output by the third layer channel branch, and the feature matrix output by the fourth layer channel branch to obtain a first concatenated result, and using a convolutional layer to assign weights to the channel dimension of the first concatenated result to obtain an output feature matrix of the first part structure of the feature enhancement module;
[0072] S104, performing global average pooling and maximum pooling on the channel dimension of the output feature matrix of the first part of the feature enhancement module structure in sequence to obtain pooled features, performing splicing and convolution activation on the pooled features along the channel dimension in sequence to obtain weight matrices for each spatial position of the input feature matrix, and multiplying the weight matrices for each spatial position of the input feature matrix by the output feature matrix of the first part of the feature enhancement module structure to obtain feature matrices output by the spatial branches of the first layer and matrices output by the skip connections of the first layer;
[0073] S105. Taking the matrix output by the skip connection of the first layer as input and iteratively repeating step S104, respectively obtain the feature matrix of the spatial branch output of the second layer, the feature matrix of the spatial branch output of the third layer, and the feature matrix of the spatial branch output of the fourth layer;
[0074] S106: Concatenate the feature matrix output by the spatial branch of the first layer, the feature matrix output by the spatial branch of the second layer, the feature matrix output by the spatial branch of the third layer, and the feature matrix output by the spatial branch of the fourth layer along the channel dimension to obtain a second concatenated result, and perform channel dimension weighting on the second concatenated result through a convolutional layer to obtain an output of the second part modeling spatial position dependency.
[0075] S107: Add the output of the second part modeling spatial position dependency and the output of the feature fusion module to obtain the output of the feature enhancement module.
[0076] The spatial dimension of the output of the feature fusion module is processed by global average pooling using the feature enhancement module to obtain the result after global average pooling. The result after global average pooling is processed by channel allocation weight through the convolution layer to obtain the channel weight matrix of the input feature. The channel weight matrix of the input feature is multiplied by the original matrix output by the feature fusion module to obtain the feature matrix of the channel branch output of the first layer and the matrix of the jump connection output of the first layer. The relationship between the corresponding processes is as follows:
[0077] ;
[0078] in, Represents the feature matrix of the channel branch output of the first layer, represents channel attention calculation, The matrix representing the output of the first layer of skip connection in the first part of the structure;
[0079] In the steps of iteratively repeating step S101 by taking the matrix output by the first layer skip connection as input to obtain the feature matrix of the second layer channel branch output, the feature matrix of the third layer channel branch output, and the feature matrix of the fourth layer channel branch output, the corresponding process has the following relationship:
[0080] ;
[0081] in, Represents the feature matrix of the channel branch output of the second layer, Represents the matrix of the output of the second layer of skip connection in the first part of the structure, Represents the feature matrix of the channel branch output of the third layer, Represents the matrix of the output of the third layer of skip connection in the first part of the structure, Represents the feature matrix of the channel branch output of the fourth layer;
[0082] In the step of concatenating the feature matrix output by the first-layer channel branch, the feature matrix output by the second-layer channel branch, the feature matrix output by the third-layer channel branch, and the feature matrix output by the fourth-layer channel branch to obtain a first concatenated result, and assigning weights to the channel dimension of the first concatenated result using a convolutional layer to obtain the output feature matrix of the first part of the feature enhancement module structure, the corresponding process has the following relationship:
[0083] ;
[0084] in, Represents the output feature matrix of the first part of the feature enhancement module.
[0085] The channel dimension of the output feature matrix of the first part of the feature enhancement module is sequentially subjected to global average pooling and maximum pooling to obtain the pooled features. The pooled features are sequentially subjected to splicing and convolution activation along the channel dimension to obtain the weight matrix of each spatial position of the input feature matrix. The weight matrix of each spatial position of the input feature matrix is multiplied by the output feature matrix of the first part of the feature enhancement module to obtain the feature matrix output by the spatial branch of the first layer and the matrix output by the jump connection of the first layer. The relationship between the corresponding processes is as follows:
[0086] ;
[0087] in, represents the feature matrix of the spatial branch output of the first layer, represents channel attention calculation, The matrix representing the output of the first layer of skip connection in the second part of the structure;
[0088] In the step of taking the matrix output by the skip connection of the first layer as input and iteratively repeating step S104 to obtain the feature matrix of the spatial branch output of the second layer, the feature matrix of the spatial branch output of the third layer, and the feature matrix of the spatial branch output of the fourth layer, the corresponding process has the following relationship:
[0089] ;
[0090] in, Represents the feature matrix of the spatial branch output of the second layer, Represents the matrix of the second layer skip connection output of the second part structure, Represents the feature matrix of the spatial branch output of the third layer, Represents the matrix of the third layer skip connection output of the second part structure, Represents the feature matrix of the spatial branch output of the fourth layer;
[0091] In the step of concatenating the feature matrix output by the spatial branch of the first layer, the feature matrix output by the spatial branch of the second layer, the feature matrix output by the spatial branch of the third layer, and the feature matrix output by the spatial branch of the fourth layer along the channel dimension to obtain the second concatenated result, and performing channel dimension weighting processing on the second concatenated result through the convolution layer to obtain the output of the second part of modeling the spatial position dependency, the corresponding process has the following relationship:
[0092] ;
[0093] in, represents the output of the second part of modeling spatial location dependencies;
[0094] In the step of adding the output of the spatial position dependency modeling module in the second part to the output of the feature fusion module to obtain the output of the feature enhancement module, the corresponding process has the following relationship:
[0095] ;
[0096] in, Represents the output of the feature boosting module.
[0097] Step 5: Pre-train the optimized tracking model using a large-scale data set, and update the parameters of the tracking model to obtain a tracking model with updated parameters.
[0098] The network was further trained using the training set of the GOT-10K, LaSOT, COCO2017, and TrackingNet datasets. 60,000 image pairs were randomly sampled from the video frames in the training set and trained for 300 epochs. The batch size was set to 32 and the learning rate was set to e-4. The learning rate was reduced after 240 epochs to prevent overfitting.
[0099] Step 6: Input the search image and template image into the tracking model with updated parameters, repeat steps 2 to 4 in an iterative manner to obtain the final enhanced features, and input the final enhanced features into the prediction head to obtain the tracking results.
[0100] In step 6, the search image and template image are input into the tracking model with updated parameters, and steps 1 to 4 are repeated iteratively to obtain the final enhanced features. The corresponding relationship is as follows:
[0101] ;
[0102] in, Represents the template features after feature extraction, Represents the search area features after feature extraction, Represents the features of the template features and the search area features spliced along the channel dimension, Representing template features length, represents the image state space module, represents the feature fusion module, Represents the feature enhancement module.
[0103] Furthermore, the present invention selects a Mamba visual mixing module for feature extraction; uses a feature aggregation module to perform feature fusion to reduce the influence of irrelevant features on tracking results; designs the network as a dual-stream structure to extract features from the template and the search area respectively; uses a feature enhancement module to further process the fused features, and enhances the relevant feature areas of the target object through multi-layer modeling channels and spatial dependencies; pre-trains a network model based on the Mamba visual mixing module, and uses the pre-trained network model to first extract features from the linearly embedded template image, and then extract features from the search area image; connects the template features and the last layer of search area features to perform multi-layer feature interaction, uses the image state space module to process the search area features, integrates the historical state information of the search area image, and fuses these features with the interacted features through the feature aggregation module; selectively enhances the relevant features of the target object through the feature enhancement module after the fusion, and then sends it to the prediction head to obtain the maximum response area of the tracking target in the current frame and mark it. The present invention fully combines the advantages of Mamba and Transformer to construct a network model based on the Mamba visual hybrid module, and uses the feature aggregation module and feature enhancement module to improve the performance of the model, so that the tracker has faster tracking speed and stronger generalization ability, can meet the needs of different tracking scenarios, and can achieve more accurate and robust tracking.
[0104] Please refer to Figure 5 The present invention also proposes a target tracking system based on the Mamba visual hybrid module, the system comprising:
[0105] Building blocks for:
[0106] The Mamba visual hybrid module is built based on the Transformer architecture and the Mamba state space model. The feature fusion module is built based on the feature aggregation mechanism. The feature enhancement module is built based on the feature enhancement mechanism. The Mamba visual hybrid module, feature fusion module, and feature enhancement module constitute the tracking model. The Mamba visual hybrid module includes multi-head self-attention and image state space modules.
[0107] Extraction module for:
[0108] The search image and template image are input into the Mamba visual mixing module, global context information and local detail information are extracted and superimposed, and the output of the Mamba visual mixing module is obtained;
[0109] Compute module for:
[0110] The output of the Mamba visual mixing module is input into the feature fusion module for fusion to obtain the output of the feature fusion module;
[0111] Learning modules for:
[0112] The output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The enhanced features are used to calculate the classification loss and regression loss respectively. The classification loss and regression loss are used to optimize the tracking model to obtain the optimized tracking model.
[0113] Pre-training modules for:
[0114] Pre-training the optimized tracking model using a large-scale dataset and updating the parameters of the tracking model to obtain a tracking model with updated parameters;
[0115] Tracking module for:
[0116] The search image and template image are input into the tracking model with updated parameters. The global context information and local detail information are extracted and superimposed in an iterative manner using the Mamba visual mixing module, and fused through the feature fusion module. Finally, the feature enhancement module is used to enhance the features to obtain the final enhanced features. The final enhanced features are input into the prediction head to obtain the tracking results.
[0117] It should be understood that various components of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.
[0118] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
[0119] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.
Claims
1. A target tracking method based on the Mamba visual hybrid module, characterized in that: The method comprises the following steps: Step 1: Construct a Mamba visual hybrid module based on the Transformer architecture and the Mamba state space model, a feature fusion module based on the feature aggregation mechanism, and a feature enhancement module based on the feature enhancement mechanism. The Mamba visual hybrid module, the feature fusion module, and the feature enhancement module constitute the tracking model; the Mamba visual hybrid module includes a multi-head self-attention and image state space module; Step 2: Input the search image and template image into the Mamba visual mixing module, extract and superimpose global context information and local detail information, and obtain the output of the Mamba visual mixing module; Step 3: Input the output of the Mamba visual mixing module into the feature fusion module for fusion to obtain the output of the feature fusion module; Step 4: The output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The enhanced features are used to calculate the classification loss and regression loss respectively. The classification loss and regression loss are used to optimize the tracking model to obtain the optimized tracking model. Step 5: Pre-train the optimized tracking model using a large-scale data set, and update the parameters of the tracking model to obtain a tracking model with updated parameters; Step 6: Input the search image and template image into the tracking model with updated parameters, repeat steps 2 to 4 in an iterative manner to obtain the final enhanced features, and input the final enhanced features into the prediction head to obtain the tracking results; In step 2, the search image and the template image are input into the Mamba visual mixing module, and global context information and local detail information are extracted and superimposed to obtain the output of the Mamba visual mixing module. Specifically, the steps include: Input the search image and template image into the Mamba visual mixing module for initialization processing respectively, and obtain the image sequence of the template area and the image sequence of the search area; The image sequence of the template area and the image sequence of the search area are respectively subjected to feature extraction using a convolutional network to obtain the original features of the image sequence of the template area and the original features of the image sequence of the search area; The original features of the template region image sequence are flattened and position-encoded in sequence to obtain the original feature label sequence of the template image; The original features of the search area image sequence are flattened and position-coded in sequence to obtain the original feature label sequence of the search area image; The original feature tag sequence of the template image is input into the multi-head self-attention to extract information and obtain the global context features of the image; The original feature label sequence of the search area image is input into the image state space module to extract local information and obtain the local features of the image; The global context features of the image are superimposed with the local features of the image to obtain the output of the Mamba visual mixing module; In step 4, the output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features, specifically in the following steps: S101, perform global average pooling processing on the spatial dimension of the output of the feature fusion module using the feature enhancement module to obtain the result after global average pooling processing, perform channel allocation weight processing on the result after global average pooling processing through the convolution layer to obtain the channel weight matrix of the input feature, multiply the channel weight matrix of the input feature with the original matrix output by the feature fusion module to obtain the feature matrix of the first layer channel branch output and the matrix of the first layer skip connection output. The relationship between the corresponding processes is as follows: ； in, Represents the feature matrix of the channel branch output of the first layer, represents channel attention calculation, The matrix representing the output of the first layer of skip connection in the first part of the structure; S102, using the matrix output by the first layer of skip connections as input to iteratively repeat step S101, and respectively obtain the feature matrix of the second layer channel branch output, the feature matrix of the third layer channel branch output, and the feature matrix of the fourth layer channel branch output. The corresponding relationship in the process is as follows: ； in, Represents the feature matrix of the channel branch output of the second layer, Represents the matrix of the output of the second layer of skip connection in the first part of the structure, Represents the feature matrix of the channel branch output of the third layer, Represents the matrix of the output of the third layer of skip connection in the first part of the structure, Represents the feature matrix of the channel branch output of the fourth layer; S103, concatenate the feature matrix output by the first layer channel branch, the feature matrix output by the second layer channel branch, the feature matrix output by the third layer channel branch, and the feature matrix output by the fourth layer channel branch to obtain a first concatenated result, and use a convolutional layer to assign weights to the channel dimension of the first concatenated result to obtain an output feature matrix of the first part structure of the feature enhancement module. The corresponding process has the following relationship: ； in, The output feature matrix representing the first part of the feature enhancement module; S104, performing global average pooling and maximum pooling processing on the channel dimension of the output feature matrix of the first part of the feature enhancement module structure in sequence to obtain pooled features, performing splicing processing and convolution activation processing on the pooled features along the channel dimension in sequence to obtain the weight matrix of each spatial position of the input feature matrix, multiplying the weight matrix of each spatial position of the input feature matrix with the output feature matrix of the first part of the feature enhancement module structure to obtain the feature matrix output by the spatial branch of the first layer and the matrix output by the skip connection of the first layer. The relationship between the corresponding processes is as follows: ； in, represents the feature matrix of the spatial branch output of the first layer, represents channel attention calculation, The matrix representing the output of the first layer of skip connection in the second part of the structure; S105. The matrix output by the first layer of skip connections is used as input and step S104 is repeated iteratively to obtain the feature matrix of the spatial branch output of the second layer, the feature matrix of the spatial branch output of the third layer, and the feature matrix of the spatial branch output of the fourth layer. The relationship between the corresponding processes is as follows: ； in, Represents the feature matrix of the spatial branch output of the second layer, Represents the matrix of the second layer skip connection output of the second part structure, Represents the feature matrix of the spatial branch output of the third layer, Represents the matrix of the third layer skip connection output of the second part structure, Represents the feature matrix of the spatial branch output of the fourth layer; S106. The feature matrix output by the spatial branch of the first layer, the feature matrix output by the spatial branch of the second layer, the feature matrix output by the spatial branch of the third layer, and the feature matrix output by the spatial branch of the fourth layer are concatenated along the channel dimension to obtain a second concatenated result. The second concatenated result is subjected to channel dimension weighting processing by a convolutional layer to obtain the output of the second part modeling spatial position dependency. The corresponding process has the following relationship: ； in, represents the output of the second part of modeling spatial location dependencies; S107: Add the output of the spatial position dependency modeling in the second part to the output of the feature fusion module to obtain the output of the feature enhancement module. The corresponding relationship is as follows: ； in, Represents the output of the feature boosting module.
2. The target tracking method based on the Mamba visual hybrid module according to claim 1 is characterized in that: The original feature tag sequence of the template image is input into the multi-head self-attention for information extraction. The corresponding process has the following relationship: ； in, represents the middle value, represents attention calculation, Represents the input of the Mamba visual mixing module, represents the normalization function, represents the query vector, represents the transpose of the key vector, represents the square root of the key vector dimension, represents a value vector; In the step of inputting the original feature label sequence of the search area image into the image state space module to extract local information and obtain the local features of the image, the corresponding process has the following relationship: ； in, Representing a state-space model Current moment Input, represents the first learnable matrix, Representing a state-space model Past Moment The information set of all input images, represents the second learnable matrix, represents the third learnable matrix, represents the fourth learnable matrix, represents the matrix multiplication operation, represents the state branch of the image state space module, represents the symmetric convolution branch of the image state space module, Indicates the first linear mapping process, represents the output of the state branch of the image state space module, Indicates activation operation. represents a one-dimensional ordinary convolution operation, Indicates the second linear mapping process, Indicates a connection operation; In the step of superimposing the global context features of the image with the local features of the image to obtain the output of the Mamba visual mixing module, the corresponding process has the following relationship: ； in, Represents the output of the Mamba visual mixing module.
3. The target tracking method based on the Mamba visual hybrid module according to claim 2 is characterized in that: In step 3, the output of the Mamba visual mixing module is input into the feature fusion module for fusion to obtain the output of the feature fusion module, which specifically includes the following steps: The image state space module is used to integrate the historical state information of the search area to obtain the integrated historical state information of the search area; The output of the Mamba visual mixing module is input into the feature fusion module, and the splicing operation, convolution activation operation and residual connection operation are performed in sequence with the historical state information of the integrated search area to obtain the output of the feature fusion module.
4. The target tracking method based on the Mamba visual hybrid module according to claim 3 is characterized in that: The output of the Mamba visual mixing module is input into the feature fusion module, and the historical state information of the integrated search area is sequentially subjected to splicing operations, convolution activation operations, and residual connection operations to obtain the output of the feature fusion module. The corresponding process has the following relationship: ； in, Represents the features output after concatenation and linear mapping, represents the fully connected layer, Represents the features after the interaction between the template area and the search area, represents the features output by the image state space module, Indicates fusion processing, represents the batch normalization operation, Represents a convolution operation with a convolution kernel size of 1×1, represents the activation function, Represents the output of the feature fusion module.
5. The target tracking method based on the Mamba visual hybrid module according to claim 1 is characterized in that: In step 6, the search image and the template image are input into the tracking model after parameter update, and steps 1 to 4 are repeated in an iterative manner to obtain the final enhanced features. The relationship between the corresponding process is as follows: ； in, Represents the template features after feature extraction, Represents the search area features after feature extraction, Represents the features of the template features and the search area features spliced along the channel dimension, Representing template features length, represents the image state space module, represents the feature fusion module, Represents the feature enhancement module.
6. A target tracking system based on the Mamba visual hybrid module, characterized in that: The system applies the target tracking method based on the Mamba visual hybrid module as described in any one of claims 1 to 5 above, and the system includes: Building blocks for: The Mamba visual hybrid module is built based on the Transformer architecture and the Mamba state space model. The feature fusion module is built based on the feature aggregation mechanism. The feature enhancement module is built based on the feature enhancement mechanism. The Mamba visual hybrid module, feature fusion module, and feature enhancement module constitute the tracking model. The Mamba visual hybrid module includes multi-head self-attention and image state space modules. Extraction module for: The search image and template image are input into the Mamba visual mixing module, global context information and local detail information are extracted and superimposed, and the output of the Mamba visual mixing module is obtained; Compute module for: The output of the Mamba visual mixing module is input into the feature fusion module for fusion to obtain the output of the feature fusion module; Learning modules for: The output of the feature fusion module is enhanced by the feature enhancement module to obtain enhanced features. The enhanced features are used to calculate the classification loss and regression loss respectively. The classification loss and regression loss are used to optimize the tracking model to obtain the optimized tracking model. Pre-training modules for: Pre-training the optimized tracking model using a large-scale dataset and updating the parameters of the tracking model to obtain a tracking model with updated parameters; Tracking module for: The search image and template image are input into the tracking model with updated parameters. The global context information and local detail information are extracted and superimposed in an iterative manner using the Mamba visual mixing module, and fused through the feature fusion module. Finally, the feature enhancement module is used to enhance the features to obtain the final enhanced features. The final enhanced features are input into the prediction head to obtain the tracking results.