Multi-modal target tracking method based on coupling-decoupling feature enhancement

Abstract

The invention discloses a multi-modal target tracking method based on coupling-decoupling feature enhancement. In order to solve the problems of modal difference, semantic conflict, insufficient utilization of complementary information and the like between infrared and visible light images, a coupling-decoupling feature enhancement module is provided to improve the expression ability and discrimination of multi-modal image fusion features. In the coupling stage, multi-modal target information is effectively fused through cross-modal cross attention, and mixed search features containing contexts are generated; in a decoupling stage, modal perception attention is performed on each modal template so as to adaptively obtain specific complementary information, and semantic consistency is kept. And then, in-mold interaction is used for enhancing discriminant features of a search area, so that the positioning accuracy is improved. According to the method, the complementary advantages of multi-modal data can be fully played in a complex environment, the robustness and precision of a target tracking system are improved, and the method is suitable for various visual tasks of multi-sensor fusion.

Description

Technical Field

[0001] The present invention relates to the fields of image processing and target tracking, and in particular to a multimodal target tracking method based on coupling-decoupling interaction. Background Art

[0002] Multimodal visual tracking, as an important branch of target tracking, plays a key role in practical applications such as security surveillance, autonomous driving, and intelligent perception. It holds broad application prospects and significant research significance. This task aims to leverage image information from multiple modalities (such as infrared and visible light) to achieve robust target tracking, addressing the poor performance of a single modality in complex scenarios characterized by varying illumination, occlusion, and background interference. Infrared images provide stable information about target thermal radiation in low or even no illumination conditions, while visible light images are more adept at capturing detailed textures and structural contours. The complementary perception capabilities of the two modalities enable the construction of more reliable tracking systems. In recent years, with the rapid development of sensor technology and significant increases in computing power, multimodal visual tracking technology has gradually transitioned from theoretical research to practical applications, demonstrating significant potential in key areas such as military reconnaissance, intelligent transportation, and industrial inspection.

[0003] Due to the significant differences between infrared and visible light in imaging mechanisms, semantic distribution, and statistical characteristics, information fusion between modalities often leads to increased redundancy, semantic mismatch, and even the submergence of target features. Specifically, these differences are mainly reflected in the following aspects: First, in terms of imaging principles, infrared images reflect the radiation characteristics of the target, while visible light images reflect the reflective characteristics of the target. This leads to the fact that the same target may appear completely different in different modalities. Second, in terms of feature distribution, the statistical characteristics of the two modalities differ significantly, and direct fusion can easily lead to mismatches in feature space. Third, in terms of semantic expression, different modalities may have different understandings of the scene, especially in scenes with complex backgrounds, drastic lighting changes, or weak target features. These differences are even more pronounced. These factors pose a huge challenge to the effective fusion of multimodal information.

[0004] Currently, most multimodal fusion methods rely on attention mechanisms or weighted fusion strategies to uniformly model and integrate information from different modalities. Although this improves feature representation capabilities to a certain extent, it still faces problems such as low fusion efficiency and poor adaptability when dealing with high-dynamic background interference, extremely low signal-to-noise ratios, or severe occlusions. Therefore, it is urgent to study a multimodal information interaction mechanism that can effectively adapt to modal differences and improve the efficiency of modal complementarity, so as to enhance the discriminative ability and robustness of tracking models in complex environments. Summary of the Invention

[0005] To address the above technical problems, the present invention proposes a multimodal target tracking method based on coupled-decoupled feature enhancement, which achieves optimal fusion of cross-modal features through a structured two-stage process. In the coupling stage, a bidirectional cross-attention mechanism is used to establish cross-modal feature interaction, and the attention outputs of the visible light and infrared modalities are fused into a unified hybrid feature representation through a linear transformation layer. In the decoupling stage, a modality-specific feature refinement unit is designed. The hybrid feature is first used as a global context to interact with the template features of each modality to generate an enhanced template. Subsequently, the enhanced template features are fed back to the original search area through intra-modal interaction to complete the discriminant feature optimization.

[0006] The technical solution adopted by the present invention is: a multimodal target tracking method based on coupling-decoupling feature enhancement, which has the following steps:

[0007] Step 1: Obtain template and search area images for visible light and infrared modalities , and extract the corresponding initial eigenvector ;

[0008] Step 2: Perform joint feature extraction and relationship modeling on each modality template and search area through a multi-layer encoder;

[0009] Step 3: Input the features output by the encoder into the coupling module and use target-aware cross-modal attention to couple the multimodal information into a hybrid search feature. middle;

[0010] Step 4: In the decoupling phase, use templates to search for features from the mixture Decouple the complementary features of modal perception and use them to enhance the discriminability of each modal search area;

[0011] Step 5: Fuse the enhanced search areas of the two modalities and input them into the head network to obtain the final tracking result.

[0012] The advantages of the present invention compared with the prior art are:

[0013] (1) The coupling-decoupling feature enhancement module proposed in this paper adopts a phased approach. In the coupling phase, bidirectional cross-attention is used to establish cross-modal associations. In the decoupling phase, modality-specific feature reconstruction is used to eliminate redundant information while preserving the unique characteristics of each modality. This structured processing method effectively solves the modal feature conflict problem existing in traditional multimodal tracking and significantly improves tracking accuracy in complex environments.

[0014] (2) The present invention can dynamically adjust the modal fusion strategy under different environmental conditions, focusing on the texture details of the visible light modality when there is sufficient light, and prioritizing the radiation characteristics of the thermal infrared modality under low-light conditions, while maintaining the specific expression of each modality. This adaptive capability enables the system to maintain stable tracking performance in complex scenes such as drastic lighting changes and partial or complete occlusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 Flowchart of the multimodal target tracking method based on coupling-decoupling feature enhancement of the present invention;

[0016] Figure 2 This is a structural diagram of the tracking network proposed in the present invention. DETAILED DESCRIPTION

[0017] In order to make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other. To achieve the above-mentioned objectives, the present invention adopts the following technical solutions.

[0018] The specific flow chart of the multimodal target tracking method based on coupling-decoupling feature enhancement of the present invention is as follows: Figure 1 As shown in the following figure, the specific network structure is as follows Figure 2 As shown. The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It includes the following steps:

[0019] Step 1: Obtain templates and search images for visible light and infrared modalities , usually the search area size is , and the size of the template is , to avoid introducing too much interference context; then divide it into non-overlapping image blocks of size , and flatten to get four image sequences. Use the image block embedding layer and add position information to get the corresponding feature vector , the specific calculation process is as follows:

[0020] (1)

[0021] The meanings of the symbols in formula (1) are:

[0022] : initial feature vector of visible light template;

[0023] : initial feature vector of visible light search area;

[0024] : initial feature vector of infrared template;

[0025] : initial feature vector of infrared search area;

[0026] :The first image blocks;

[0027] : The first visible light search area image blocks;

[0028] :Infrared template image blocks;

[0029] : The infrared search area image blocks;

[0030] : the total number of image blocks of the template;

[0031] : The total number of image blocks in the search area;

[0032] : embedding projection weight of visible light modality;

[0033] : Embedding projection weights of thermal infrared modality;

[0034] : The position of the template is embedded;

[0035] : Position embedding of the search area;

[0036] Step 2: Through two symmetrically branched multi-layer encoders, multi-head attention is used to perform joint feature extraction and relationship modeling on each modality template and search area. The parameters of the two branches are shared. The specific calculation process of the visible light branch is as follows:

[0037] (2)

[0038] The meanings of the symbols in formula (2) are:

[0039] : No. Search area features input to the layer encoder;

[0040] : No. Template features input to the layer encoder;

[0041] : Feature representation after weighted aggregation using attention matrix;

[0042] : Search area features after preliminary enhancement;

[0043] : Template features after preliminary enhancement;

[0044] : No. Learnable query weight matrix in the layer encoder self-attention process;

[0045] : No. The learnable key weight matrix in the layer encoder self-attention process;

[0046] : No. The learnable value weight matrix of the layer encoder self-attention process;

[0047] : scaling factor;

[0048] : normalized exponential function;

[0049] : layer normalization;

[0050] : Multilayer Perceptron;

[0051] Step 3: Input the features output by the encoder into the coupling module, use target-aware cross-modal attention, use the search area as the query vector, and use the cross-modal template as the key vector to couple the multimodal information into a hybrid search feature. The calculation process is as follows:

[0052] (3)

[0053] The meanings of the symbols in formula (3) are:

[0054] : No. Learnable query weight matrix in sub-cross attention;

[0055] : No. Learnable key weight matrix in sub-cross attention;

[0056] : No. Learnable value weight matrix in sub-cross attention;

[0057] : Infrared target features after weighted aggregation in the coupling stage;

[0058] : Visible light target features after weighted aggregation in the coupling stage;

[0059] : Visible light search area features enhanced with infrared target information;

[0060] : Infrared search area features enhanced with visible light target information;

[0061] : Refined visible light search area features;

[0062] : Refined infrared search area features;

[0063] : Hybrid search area features;

[0064] : Linear projection layer;

[0065] Step 4: In the decoupling phase, use templates to search for features from the mixture The complementary features of modal perception are decoupled and used to enhance the discriminability of each modal search area. The specific steps are as follows:

[0066] Step 4a: Use each modal template as a query vector to adaptively decouple the complementary information in the hybrid search area. This process can adaptively control the sharing of each modality based on the relationship between each token of the template and the two modalities, and is used to enhance the encoding of the current template while maintaining semantic consistency. The specific calculation formula is as follows:

[0067] (4)

[0068] The meanings of the symbols in formula (4) are:

[0069] : Visible light complementary features of target-related context;

[0070] : infrared complementary features of target-related context;

[0071] : Visible light template features enhanced using complementary information;

[0072] : Infrared template features enhanced using complementary information;

[0073] Step 4b: Perform intra-modal interaction directly between the template and the search area. Since the template has been effectively enhanced in the previous process, the discriminability of each search area can be strengthened accordingly. At the same time, this intra-modal interaction can effectively avoid the reintroduction of redundant modal conflicts, thereby eliminating noise. The calculation formula is as follows:

[0074] (5)

[0075] The meanings of the symbols in formula (5) are:

[0076] : Visible light target features after weighted aggregation in the decoupling stage;

[0077] : Infrared target features after weighted aggregation in the decoupling stage;

[0078] : Visible light search area features after discriminant information enhancement;

[0079] : Infrared search area features after discriminant information enhancement;

[0080] Step 5: Fuse the enhanced search areas of the two modalities and input them into the head network to obtain the final tracking result. The specific calculation process is as follows:

[0081] Step 5a: Input the fused search area features into the head network to obtain the predicted amount required to calculate the coordinates. The calculation formula is as follows:

[0082] (6)

[0083] The meanings of the symbols in formula (5) are:

[0084] : The final output of the visible light search area features;

[0085] : The final output infrared search area features;

[0086] : The fused multimodal search area features;

[0087] : The convolutional layer used for each prediction;

[0088] : predicted classification heatmap;

[0089] : predicted offset;

[0090] : Predicted target width and height;

[0091] Step 5b: Decode the predicted amount to obtain the final predicted box coordinates. The formula is as follows:

[0092] (7)

[0093] The meanings of the symbols in formula (7) are:

[0094] : x-axis coordinate of the peak point of the heat map;

[0095] : y-axis coordinate of the heat map peak point;

[0096] : Downsampling multiple of the feature map to the original image;

[0097] : x-axis coordinate of the target center;

[0098] : The y-axis coordinate of the target center;

[0099] : Width of the target;

[0100] : target height;

[0101] The specific implementation methods of the present invention have been described in detail through the above embodiments, but the scope of protection thereof is not limited thereto. Any technical personnel in the relevant technical field, based on the technical principles and core design concepts disclosed in the present invention, shall be deemed to be included in the patent protection scope of the present invention by implementing the technical solutions through equivalent replacement, appropriate modification or reasonable expansion, as long as the technical essence thereof does not deviate from the protection scope defined by the claims of the present invention.

Claims

1. A multimodal target tracking method based on coupling-decoupling feature enhancement, characterized in that: The following steps are involved: Step 1: Obtain template and search area images for visible light and infrared modalities , and extract the corresponding initial eigenvector ; Step 2: Perform joint feature extraction and relationship modeling on each modality template and search area through a multi-layer encoder; Step 3: Input the features output by the encoder into the coupling module and use target-aware cross-modal attention to couple the multimodal information into a hybrid search feature. middle; Step 4: In the decoupling phase, use templates to search for features from the mixture Decouple the complementary features of modal perception and use them to enhance the discriminability of each modal search area; Step 5: Fuse the enhanced search areas of the two modalities and input them into the head network to obtain the final tracking result.

2. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 1, characterized in that: In step 1, templates and search area images of visible light and thermal infrared images The image is first divided into non-overlapping blocks of size , flatten it to get four image sequences, pass them through the embedding layer respectively, and add the position information to get the corresponding feature vector. The specific calculation process is as follows: (1) The meanings of the symbols in formula (1) are: : initial feature vector of visible light template; : initial feature vector of visible light search area; : initial feature vector of infrared template; : initial feature vector of infrared search area; :The first image blocks; : The first visible light search area image blocks; :Infrared template image blocks; : The infrared search area image blocks; : the total number of image blocks of the template; : The total number of image blocks in the search area; : embedding projection weight of visible light modality; : Embedded projection weights of thermal infrared modality; : The position of the template is embedded; : The location embedding of the search area.

3. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 1, characterized in that: In step 2, a multi-layer encoder with two symmetrical branches uses multi-head attention to perform joint feature extraction and relationship modeling on each modality template and search area. The specific calculation process of the visible light branch is as follows: (2) The meanings of the symbols in formula (2) are: : No. Search area features input to the layer encoder; : No. Template features input to the layer encoder; : Feature representation after weighted aggregation using attention matrix; : Search area features after preliminary enhancement; : Template features after preliminary enhancement; : No. Learnable query weight matrix in the layer encoder self-attention process; : No. The learnable key weight matrix in the layer encoder self-attention process; : No. The learnable value weight matrix of the layer encoder self-attention process; : scaling factor; : normalized exponential function; : layer normalization; : Multilayer Perceptron.

4. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 1, characterized in that: The target-aware cross-modal attention calculation process described in step 3 is as follows: (3) The meanings of the symbols in formula (3) are: : No. Learnable query weight matrix in sub-cross attention; : No. Learnable key weight matrix in sub-cross attention; : No. Learnable value weight matrix in sub-cross attention; : Infrared target features after weighted aggregation in the coupling stage; : Visible light target features after weighted aggregation in the coupling stage; : Visible light search area features enhanced with infrared target information; : Infrared search area features enhanced with visible light target information; : Refined visible light search area features; : Refined infrared search area features; : Hybrid search area features; : Linear projection layer.

5. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 1, characterized in that: The calculation process of the decoupling phase described in step 4 is as follows: Step 4a: Using each modality template as a query vector, the complementary information in the hybrid search region is decoupled and used to encode and enhance the current template while maintaining semantic consistency. Step 4b: Perform intra-modal interaction between the template and the search region to enhance the discriminability of the search region of each modality while removing noise.

6. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 5, characterized in that: The specific calculation formula for step 4a is as follows: (4) The meanings of the symbols in formula (4) are: : Visible light complementary features of target-related context; : infrared complementary features of target-related context; : Visible light template features enhanced using complementary information; : Infrared template features enhanced using complementary information.

7. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 5, characterized in that: The calculation formula for step 4b is as follows: (5) The meanings of the symbols in formula (5) are: : Visible light target features after weighted aggregation in the decoupling stage; : Infrared target features after weighted aggregation in the decoupling stage; : Visible light search area features after discriminant information enhancement; : Infrared search area features after discriminant information enhancement.

8. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 1, characterized in that: The calculation process of target positioning through the head network described in step 5 is as follows: Step 5a: Input the fused search area features into the head network to obtain the prediction quantity required for calculating the coordinates; Step 5b: Decode the predicted amount to obtain the final predicted box coordinates.

9. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 8, characterized in that: The calculation formula for step 5a is as follows: (6) The meanings of the symbols in formula (6) are: : The final output of the visible light search area features; : The final output infrared search area features; : The fused multimodal search area features; : The convolutional layer used for each prediction; : predicted classification heatmap; : predicted offset; : Predicted object width and height.

10. The multimodal target tracking method based on coupling-decoupling feature enhancement according to claim 8, characterized in that: The formula for step 5b is as follows: (7) The meanings of the symbols in formula (7) are: : x-axis coordinate of the peak point of the heat map; : y-axis coordinate of the heat map peak point; : Downsampling multiple of the feature map to the original image; : x-axis coordinate of the target center; : The y-axis coordinate of the target center; : Width of the target; : The height of the target.

Images