Second-order attention-based multi-modal adaptive fusion method and system

Abstract

The application provides a multi-modal adaptive fusion method based on second-order attention. The system comprises the following steps: introducing a second-order cross-attention in an attention block to realize multi-modal attention mutual enhancement, and taking different inputs for different regions of attention, and also considering the horizontally adjacent regions; when screening background tokens, using the attention weights of two modal branches to sort all tokens, and eliminating background tokens according to a fixed proportion. The application solves the technical problems of low inference speed of the framework model caused by large calculation amount of the feature fusion algorithm, and poor stability of the candidate elimination effect.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a multimodal adaptive fusion method and system based on second-order attention. Background Technology

[0002] Visual target tracking is one of the fundamental tasks of computer vision, with wide applications in robotics, security, and autonomous driving. However, due to limitations in data quality, current tracking methods based on visible light data are prone to failure under conditions such as changes in lighting and hazardous weather. In contrast, thermal infrared (TIR) ​​modes are almost unaffected by lighting conditions and fog, thus overcoming the shortcomings of visible light modes in this regard. Therefore, fully utilizing the modal information from both modes is key to achieving more stable tracking results.

[0003] Most existing RGBT tracking methods use MDNet as the baseline tracker and VGG-M as the network backbone. Xiao et al., in their paper "Attribute-based Progressive Fusion Network for RGBT Tracking" (Association for the Advancement of Artificial Intelligence), proposed a structure that decouples challenge attributes and uses a challenge attribute aggregation module to aggregate features from different challenge attribute branches. Finally, an enhancement fusion module further enhances the modality-specific features and attribute aggregation features. In recent years, multimodal trackers based on ViT have also emerged. Hui et al., in their paper "Bridging Search Region Interaction with Template for RGB-T Tracking" (IEEE Conference on Computer Vision and Pattern Recognition), used ViT as the backbone network. Considering that direct concatenation is too coarse and easily introduces redundant background noise, they also proposed a bridging mechanism that uses the fused template features as an intermediate medium for token interaction between different modalities, thereby reducing noise information involved in the interaction.

[0004] For example, the existing invention patent application document CN110807795A, entitled "A Method and Device for Remote Sensing Target Tracking of Unmanned Aerial Vehicles Based on MDnet," describes a method that includes: obtaining the target bounding box and corresponding target score of a video frame by passing it through a preset tracking model; wherein the tracking model is constructed based on an MDnet neural network model; according to a preset update strategy, if tracking failure is determined and the target score is within a preset fuzzy interval, the tracking model is updated according to a preset adaptive context-aware correlation filter. This embodiment of the invention performs the tracking task by constructing an MDnet-based tracking model and updates the tracking model using a preset adaptive context-aware correlation filter when the target score is within a preset fuzzy interval. However, the aforementioned existing methods based on the MDNet framework have a small number of trainable parameters and limited model scalability, resulting in limited benefits when training on large-scale datasets. In recent years, with the emergence of the large-scale public dataset LasHeR, these methods have lost their performance advantage. Furthermore, due to their reliance on numerous online update operations to fine-tune test sequences, they often have slow inference speeds, making them difficult to apply in practice.

[0005] And the existing invention patent application document CN116051602A, entitled "A Single-Target Tracking Method Based on a Graph Association-Based Candidate Elimination Module," describes a method where the network structure of the graph association-based candidate elimination module consists of a multi-head attention module and a graph association module. The multi-head attention module uses a multi-head attention mechanism to learn the correlation between different blocks. The graph association module is combined with a ViT model for single-target tracking. The graph association module performs graph structure correlation calculations between the remaining search candidate block features and template block features, establishing an explicit influence weight matrix between nodes, and associating target features and interfering object features in the occluded scene. However, while these ViT-based methods aim to fully utilize cross-attention for feature interaction, they neglect the consistency of self-attention calculations between different modalities. This consistency stems from the spatial alignment of the two modal images. Furthermore, feature-oriented cross-attention calculations impose a heavy computational burden, which is detrimental to the real-time requirements of target tracking tasks.

[0006] In summary, existing technologies suffer from technical problems such as high computational cost of feature fusion algorithms leading to low inference speed of the framework model, and poor stability of candidate elimination performance. Summary of the Invention

[0007] The technical problem to be solved by this invention is: how to solve the technical problems of the large amount of computation in the feature fusion algorithm in the prior art, which leads to the low inference speed of the framework model, and the poor stability of the candidate elimination effect.

[0008] This invention solves the above-mentioned technical problems by employing the following technical solution: a multimodal adaptive fusion method based on second-order attention, comprising:

[0009] S1. Acquire and process the initial image features. Use the second-order attention multimodal ViT mutual enhancement fusion module as the backbone of the multimodal tracker to perform feature enhancement and fusion operations on the initial image features. Based on the attention weight-oriented fusion mechanism, integrate the original attention weights of different modal branches and perform another attention processing to enhance the original attention distribution, so as to embed the modal fusion process into the modal feature extraction process.

[0010] S2. Using the unified candidate elimination module, all fused features are sorted by integrating multimodal attention weights, and subsequent features are screened out according to a predetermined ratio to obtain the multimodal feature elimination result.

[0011] This invention introduces information from another modality in the second Attention operation to help enhance the original attention distribution. Since this invention does not directly fuse features, the attention weights, when used as features, have a much smaller dimensionality than appearance features, effectively avoiding the computational burden of feature fusion.

[0012] This invention proposes Unified candidate elimination, which combines the attention weights of two modalities for judgment. This can improve the stability of token elimination when the quality of a single modality deteriorates, and improve the inference speed of the network model while taking into account the robustness of the system.

[0013] In a more specific technical solution, step S1 includes:

[0014] S11. Given RGB and TIR template image pairs and RGB and TIR search region image pairs, divide the images in the RGB and TIR template image pairs and RGB and TIR search region image pairs into image blocks; obtain a block embedding layer to process the image sequence and obtain the initial image features of the RGB and TIR template image pairs and RGB and TIR search region image pairs.

[0015] S12. The initial image features are processed in series to obtain the image series features. The series image features are then input into the backbone network for forward propagation. The backbone network includes: a transformer block and an acaformer block of layer L.

[0016] S13. Perform a general attention process in the transformer to extract specific modality features;

[0017] S14. In the acaformer block, compared with the general attention process, an additional AIE module is used. It treats the attention weights generated by the general attention process in different branches as weight features, thereby mining the consistency data of attention weights in different modal branches. On the original attention weights, a second attention operation is performed to enhance the original attention distribution, thereby obtaining the comprehensive multimodal attention weights. Among them, AIE processes the current adjustment region and its horizontally adjacent regions in the attention map, thereby obtaining attention mutual enhancement processing data, and thereby obtaining fusion features.

[0018] S15. Divide the features output by the backbone network into search region features and template features. Concatenate the search region features of the two modalities and input them into the tracking head to obtain the final tracking results, including the position and size of the target.

[0019] This invention employs the ViT framework, which can change the model's parameter size by altering the number of stacked blocks to adapt to different devices and fully leverage large-scale datasets to achieve performance improvements.

[0020] In the softmax operation, adjacent regions influence each other in the horizontal direction of the attention map. Therefore, when adjusting one region, this invention takes into account its horizontally adjacent regions. The Attention-Cross-Attention module of this invention introduces second-order attention on a general attention block to achieve multimodal attention mutual enhancement. At the same time, different inputs are taken for different regions of attention, and the horizontally adjacent regions are also taken into consideration, which makes the model more accurate.

[0021] In a more specific technical solution, in step S11, a learnable positional encoding is added. and The prior location information can be obtained by using the following logic:

[0022]

[0023] Wherein, the subscript r represents RGB, t represents TIR, the upper right corner subscript, z represents the template, and x represents the search area; specifically, the number represents which block layer's output this is, and since the initial token has not passed through any block, it is 0.

[0024] In a more specific technical solution, in step S12, the following logic is used to perform a forward propagation operation on the concatenated image features to obtain a multimodal tracker:

[0025]

[0026] In the formula, The output of the last transformer block is φ, which uses the same box head as the tracking head in OSTrack (cited from Ye et al., Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework, ECCV 2022).

[0027] In a more specific technical solution, in step S14, an attention mutual enhancement module AIE is embedded in the second-order attention multimodal ViT mutual enhancement fusion module ACA to perform joint optimization operations on the attention weights of different modal branches.

[0028] This invention employs a second-order attention mechanism instead of the traditional feature-oriented fusion method, treating attention weights as features and exploring the consistency of attention weights across different modal branches. Since the dimensionality of attention weights as features is much smaller than that of appearance features, it avoids the heavy computational burden of conventional feature fusion methods, resulting in inference speeds far exceeding those of other models within the same framework.

[0029] In a more specific technical solution, in step S14, the following logic is used to generate an attention map A for the RGB features. r :

[0030]

[0031] [M′ r M′ t ] = AIE(M r M t (4)

[0032]

[0033] Specifically, the Transformer's attention mechanism has three linear layers that project the input to obtain query, key, and value, with parameters W and W respectively. q W k W v .

[0034] Q r =H r W q K r =H r Wk ， Q t K t Similarly.

[0035] For Q rThe template part and the search area part, when connected together, form Q. r The same applies to K.

[0036] In a more specific technical solution, step S14 involves processing the regions in the attention map using the following logic. This region represents the association between template features and search region features:

[0037]

[0038] In the formula, LN represents LayerNorm, W e It is a learnable linear projection. and Considering the existence of the softmax operation, the horizontally adjacent regions in the attention graph affect each other. Here, both of these are used as inputs to AIE.

[0039] Where, let Q′=EW′ q K′=EW′ k ,have:

[0040]

[0041]

[0042]

[0043] Specifically, since AIE also uses the Transformer attention mechanism and also has Q, K, and V, the superscript ' is used to distinguish this part, indicating that it is a variable calculated by the AIE module.

[0044] in:

[0045]

[0046]

[0047] W′: is a learnable matrix in a linear projection layer set by AIE.

[0048] The Attention-Cross-Attention approach proposed in this invention links the self-attention processes of two branches and embeds an Attention Interaction Enhancement (AIE) module to jointly optimize the attention weights of multimodal attention. The AIE module used in this invention is also applicable to optimizing other regions of the attention map.

[0049] In a more specific technical solution, in step S2, consistent weights are assigned to the corresponding positions of images of different modalities.

[0050] In this invention, the corresponding positions of different modal images are assigned consistent weights to ensure that the final elimination of different modalities is consistent.

[0051] In a more specific technical solution, step S2 includes:

[0052] S21. Given a query vector Among them, the two query vectors are the query vectors corresponding to the template center tokens of the two modalities;

[0053] S22. Select the query vector corresponding to the token at the center of the template and assign a scalar to the token in the RGB modal search region using the following logic. Assign scalars to the tokens in the TIR modal search region

[0054]

[0055] In the formula, and It is the key vector of the search area token;

[0056] S23, Using the first scalar Second scalar The tokens in the search region are sorted, and the preceding tokens are retained. This improves the stability of token elimination when the quality of a single modality deteriorates, thereby increasing the inference speed of the network model while ensuring system robustness.

[0057] This invention also improves the candidate elimination method. When filtering background tokens, it uses the attention weights of two modal branches to sort all tokens and eliminates background tokens according to a fixed ratio. This invention employs an improved candidate elimination method, utilizing multimodal attention to make the candidate elimination effect more stable, improving both performance and efficiency, further enhancing network efficiency while maintaining network accuracy. This invention sorts all features by integrating multimodal attention weights and then removes lower-ranked features according to a predetermined ratio to reduce computational load.

[0058] In more specific technical solutions, multimodal adaptive fusion systems based on second-order attention include:

[0059] The second-order attention multimodal ViT mutual enhancement fusion module is used to collect and process the initial image features. Using the second-order attention multimodal ViT mutual enhancement fusion module as the backbone of the multimodal tracker, feature fusion operation is performed on the initial image features. Based on the original attention weights, the original attention weights of different modal branches are combined according to the attention weight fusion mechanism, and then another attention processing is performed to enhance the original attention distribution, so as to embed the modality fusion process into the modality feature extraction process.

[0060] The unified candidate elimination module is used to sort all fused features by integrating multimodal attention weights, and then filter out subsequent features according to a predetermined ratio to obtain the multimodal feature elimination result.

[0061] The present invention has the following advantages over the prior art:

[0062] This invention introduces information from another modality in the second Attention operation to help enhance the original attention distribution. Since this invention does not directly fuse features, the attention weights, when used as features, have a much smaller dimensionality than appearance features, effectively avoiding the computational burden of feature fusion.

[0063] This invention proposes Unified candidate elimination, which combines the attention weights of two modalities for judgment. This can improve the stability of token elimination when the quality of a single modality deteriorates, and improve the inference speed of the network model while taking into account the robustness of the system.

[0064] This invention employs the ViT framework, which can change the model's parameter size by altering the number of stacked blocks to adapt to different devices and fully leverage large-scale datasets to achieve performance improvements.

[0065] In softmax operations, adjacent regions can influence each other in the horizontal direction of the attention map. Therefore, when adjusting one region, this invention takes into account its horizontally adjacent regions. The Attention-Cross-Attention module of this invention introduces second-order cross-attention on a general attention block to achieve multimodal attention enhancement. At the same time, different inputs are used for different regions of attention, and the horizontally adjacent regions are also taken into consideration, which makes the model more accurate.

[0066] This invention employs a second-order attention mechanism instead of the traditional feature-oriented fusion method, treating attention weights as features and exploring the consistency of attention weights across different modal branches. Since the dimensionality of attention weights as features is much smaller than that of appearance features, it avoids the heavy computational burden of conventional feature fusion methods, resulting in inference speeds far exceeding those of other models within the same framework.

[0067] The Attention-Cross-Attention approach proposed in this invention links the self-attention processes of two branches and embeds an Attention Interaction Enhancement (AIE) module to jointly optimize the attention weights of multimodal attention. The AIE module used in this invention is also applicable to optimizing other regions of the attention map.

[0068] In this invention, the corresponding positions of different modal images are assigned consistent weights to ensure that the final elimination of different modalities is consistent.

[0069] This invention also improves the candidate elimination method. When filtering background tokens, it uses the attention weights of two modal branches to sort all tokens and eliminates background tokens according to a fixed ratio. This invention employs an improved candidate elimination method, utilizing multimodal attention to make the candidate elimination effect more stable, improving both performance and efficiency, further enhancing network efficiency while maintaining network accuracy. This invention sorts all features by integrating multimodal attention weights and then removes lower-ranked features according to a predetermined ratio to reduce computational load.

[0070] This invention solves the technical problems of existing technologies, such as the large computational load of feature fusion algorithms leading to low inference speed of framework models and poor stability of candidate elimination effects. Attached Figure Description

[0071] Figure 1 This is a schematic diagram of the overall framework of ACAFormer in Embodiment 1 of the present invention;

[0072] Figure 2 This is a schematic diagram of the basic steps of the multimodal adaptive fusion method based on second-order attention in Embodiment 1 of the present invention;

[0073] Figure 3 This is a schematic diagram of the TransFormer Block and ACAFormer Block structures in Embodiment 1 of the present invention;

[0074] Figure 4 This is a schematic diagram of the ACA structure and data flow processing of the multimodal mutual enhancement fusion module based on second-order attention in Embodiment 1 of the present invention;

[0075] Figure 5 This is a schematic diagram of the Attention Interaction Enhancement Module (AIE) structure in Embodiment 1 of the present invention;

[0076] Figure 6 This is a schematic diagram of the experimental results of Embodiment 2 of the present invention;

[0077] Figure 7 This is a visual diagram of the unified candidate elimination results in Embodiment 2 of the present invention. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Example 1

[0080] like Figure 1 and Figure 2 As shown, the multimodal adaptive fusion method based on second-order attention provided by this invention includes the following basic steps:

[0081] S1. Use a multimodal mutual enhancement fusion module based on second-order attention to perform feature fusion on image features, and perform another attention operation on the original attention weights to enhance the original attention distribution and obtain the feature fusion result.

[0082] In this embodiment, the ViT framework is used, which can change the parameter size of the model by changing the number of stacked blocks;

[0083] In this embodiment, multimodal ViT is used for RGBT tracking. Given the powerful feature extraction capability of Transformer, the visual Transformer, i.e., ViT, is extended to multimodal and used as the backbone of the tracker.

[0084] In this embodiment, given the input RGB and TIR template image pairs:

[0085] ,

[0086] Search region image pairs for RGB and TIR:

[0087]

[0088] In this embodiment, these images are first divided into small blocks of size P×P; the small blocks are then flattened to obtain... and Where N z =H z W z / P 2 N x =H x W x / P 2 Indicates the number of blocks for the template and search region; will include parameters Block embedding layers are applied to these sequences to obtain initial features of the RGB and TIR search regions and templates; learnable positional encodings are incorporated. and To provide the following prior location information:

[0089]

[0090] In this embodiment, the input image is divided into blocks of size P×P, N z =H z W z / P 2 N x =H x W x / P 2 , where W is the number of blocks obtained after dividing the image into blocks. 0 : These are the learnable parameters in the Patch Embed layer. The vector group obtained after the template image is divided into blocks and flattened into one dimension represents the original image information; the actual result is in matrix form. and

[0091] In this embodiment, This represents the initial token features obtained from the template image, i.e., the features before inputting into the Transformerblock.

[0092] The same logic applies as above; it is obtained from the image of the search region.

[0093] By connecting these features together, we obtain The input is fed into the transformer block of layer L. In this embodiment, a consistent box head, denoted as φ, is used. The final forward propagation process can be represented as:

[0094]

[0095] in, This is the output of the last transformer block. At this point, we have a basic multimodal tracker.

[0096] In this embodiment, a second-order attention mechanism is used to treat attention weights as features and to mine the consistency of attention weights in different modal branches.

[0097] like Figure 3 As shown, this embodiment employs a multimodal mutual enhancement fusion module based on second-order attention, Attention-Cross-Attention (ACA). The attention mechanism is the core of the Transformer; whether the network can focus on the correct region in the image directly determines its performance. Ideally, different branches will eventually focus on similar regions, i.e., the target itself. However, in reality, due to the unstable quality of modal images, poor image quality often prevents the network from focusing on the correct region. But through the consistency of attention, it is known that the regions of interest for different modalities can refer to each other, and the two have the conditions for mutual enhancement.

[0098] like Figure 4 As shown, in response to the aforementioned situation, this embodiment employs a mechanism that performs another Attention operation on top of the original attention weights. See [link to documentation]. Figure 4 In the second Attention operation, information from another modality is introduced in this embodiment to help enhance the original attention distribution. Since this embodiment does not directly fuse features, the dimension of the attention weights as features is much smaller than that of the appearance features. This effectively avoids the computational burden brought by feature fusion. In the experiment, the proposed Attention-Cross-Attention module only increased the amount of computation by a very small amount, but achieved results comparable to the current best feature fusion method.

[0099] like Figure 3 As shown, in this embodiment, the attention mechanism is key to the Transformer, and the attention weights are intermediate results of the attention mechanism, representing the correlation between tokens. See also Figure 3 This embodiment proposes an Attention-Cross-Attention approach that combines both approaches and embeds an Attention Interaction Enhancement (AIE) module to jointly optimize multimodal attention weights. In this embodiment, see [link to documentation]. Figure 4 Attention-Cross-Attention is a symmetrical structure. Figure 4The parameters at corresponding positions on the left and right sides are shared. For multi-head attention blocks, the parameters of the AIE module are shared among parallel attention heads. It is worth noting that in this embodiment, the entire attention map is divided into four different regions. AIE adjusts one of these regions, but can be easily applied to the other regions.

[0100] In this embodiment, we will use the Attention-Cross-Attention layer i as an example to illustrate its working process. Recall the backbone of the base tracker, the input of the i-th layer is... and For simplicity, the superscript i will be omitted below. Let Q r =H r W q ,K r =H r W k Q t ,K t Similarly, since the processing of RGB features is symmetrical to that of TIR features, we will only describe the processing of RGB features below. For RGB features, the attention map A is generated. r The process can be represented as:

[0101]

[0102] [M′ r M′ t ] = AIE(M r M t (4)

[0103]

[0104] As shown in equation (3) above, M r M t Each area can be divided into four regions with different functions. The AIE module has different processing methods for different areas.

[0105] like Figure 5 As shown, in this embodiment, with The working process of the AIE module is described using an example:

[0106]

[0107] Where LN represents LayerNorm, W e It is a learnable linear projection. Due to the softmax operation, neighboring regions (e.g., and Note that the horizontal axes of the graph influence each other; therefore, when adjusting one region, we also consider its horizontally adjacent regions, which makes our model more accurate. Let Q′ = EW′ q K′=EW′ k Then we have:

[0108]

[0109]

[0110]

[0111] S2. Using the unified candidate elimination module, all features are sorted by integrating the attention weights of the multimodal approaches, and then the features ranked lower are removed according to a predetermined ratio to obtain the multimodal adaptive fusion result.

[0112] In this embodiment, a unified candidate elimination module, Unified Candidate Elimination (UCE), is used. Inspired by the aforementioned attention consistency, this embodiment also improves the candidate elimination method, which was originally proposed by Ye et al., "Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework," at the European Conference on Computer Vision. Generally, the distribution of attention weights reflects the importance of different features. In this embodiment, all features are ranked by integrating multimodal attention weights, and then features ranked lower are removed according to a predetermined ratio to reduce computational load. It is important to note that the weights assigned to corresponding positions in different modalities are consistent to ensure consistent elimination across different modalities.

[0113] In this embodiment, to improve inference speed while maintaining robustness, a unified candidate elimination method is proposed. This method combines the attention weights of the two modalities for judgment, which can improve the stability of token elimination when the quality of a single modality deteriorates. Given a query vector... In this embodiment, selecting the query vector corresponding to the template center token allows a scalar to be assigned to the token in each search region. or

[0114]

[0115] in, and This is the key vector of the search region token. Afterwards, we will use... and Sort the tokens in the search area and keep the tokens with the highest ranking.

[0116] Example 2

[0117] In the quantitative evaluation of this embodiment, visible light data and thermal infrared data are selected as the two modalities in the field of computer vision tracking. In this embodiment, the algorithm of this application is evaluated by comparing its tracking performance with some state-of-the-art trackers on four tracking datasets (RGBT210, RGBT234, LasHeR, and VTUAV) to verify the effectiveness of the proposed method. See Tables 1 and 2 below. This application uses accuracy (PR) and success rate (SR) in the overall evaluation (OPE) as evaluation indicators for quantitative performance evaluation. For RGBT234 and VTUAV, due to the characteristics of the datasets, the usual maximum accuracy (MPR) and maximum success rate (MSR) are used.

[0118] Table 1 Comparison of Method Results

[0119]

[0120] Table 2 compares the method results on the VTUAV dataset.

[0121]

[0122] In this embodiment, accuracy is the percentage of all predicted frames whose ground truth values ​​are less than a threshold, which is set to 20 pixels. Success rate is the percentage of successfully tracked frames whose overlapping portion exceeds the threshold. The success rate score is calculated by the area under the success rate curve. The maximum success rate is the largest of the two modal ground truth values, and the maximum accuracy is calculated similarly. Considering that accuracy is highly sensitive to target size, the normalized precision (NPR) obtained by normalizing the ground truth values ​​is used to evaluate the tracking performance on the LasHeR dataset. The results show that this invention significantly improves the performance in the field of computer vision fusion tracking.

[0123] like Figure 6 As shown, in the qualitative evaluation process of this embodiment, the results of the qualitative evaluation demonstrate that, considering the characteristics of the two modes, when one mode has a large amount of noise, the present invention can significantly extract effective features from the other mode, thereby enhancing the results. See also... Figure 6The top left image is the original RGB mode image, the top right image is the original T mode image, the bottom left image is the RGB mode tracking result, and the bottom right image is the T mode tracking result.

[0124] like Figure 7 As shown in this embodiment, for the unified candidate elimination module, it can be seen that even if a certain modality contains a lot of noise, it still effectively retains the token containing the target, avoiding the loss of key information and reducing the introduction of noise. See also Figure 7 The top image shows the visible light modal image, and the bottom image shows the thermal infrared modal image. From left to right, these represent the elimination results for layers 1, 4, 7, and 10 of the network.

[0125] In summary, this invention introduces information from another modality in the second Attention operation to help enhance the original attention distribution. Since this invention does not directly fuse features, the attention weights, when used as features, have a much smaller dimensionality than appearance features, effectively avoiding the computational burden of feature fusion.

[0126] This invention proposes Unified candidate elimination, which combines the attention weights of two modalities for judgment. This can improve the stability of token elimination when the quality of a single modality deteriorates, and improve the inference speed of the network model while taking into account the robustness of the system.

[0127] This invention employs the ViT framework, which can change the model's parameter size by altering the number of stacked blocks to adapt to different devices and fully leverage large-scale datasets to achieve performance improvements.

[0128] In softmax operations, adjacent regions can influence each other in the horizontal direction of the attention map. Therefore, when adjusting one region, this invention takes into account its horizontally adjacent regions. The Attention-Cross-Attention module of this invention introduces second-order cross-attention on a general attention block to achieve multimodal attention enhancement. At the same time, different inputs are used for different regions of attention, and the horizontally adjacent regions are also taken into consideration, which makes the model more accurate.

[0129] This invention employs a second-order attention mechanism instead of the traditional feature-oriented fusion method, treating attention weights as features and exploring the consistency of attention weights across different modal branches. Since the dimensionality of attention weights as features is much smaller than that of appearance features, it avoids the heavy computational burden of conventional feature fusion methods, resulting in inference speeds far exceeding those of other models within the same framework.

[0130] The Attention-Cross-Attention approach proposed in this invention links the self-attention processes of two branches and embeds an Attention Interaction Enhancement (AIE) module to jointly optimize the attention weights of multimodal attention. The AIE module used in this invention is also applicable to optimizing other regions of the attention map.

[0131] In this invention, the corresponding positions of different modal images are assigned consistent weights to ensure that the final elimination of different modalities is consistent.

[0132] This invention also improves the candidate elimination method. When filtering background tokens, it uses the attention weights of two modal branches to sort all tokens and eliminates background tokens according to a fixed ratio. This invention employs an improved candidate elimination method, utilizing multimodal attention to make the candidate elimination effect more stable, improving both performance and efficiency, further enhancing network efficiency while maintaining network accuracy. This invention sorts all features by integrating multimodal attention weights and then removes lower-ranked features according to a predetermined ratio to reduce computational load.

[0133] This invention solves the technical problems of existing technologies, such as the large computational load of feature fusion algorithms leading to low inference speed of framework models and poor stability of candidate elimination effects.

[0134] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal adaptive fusion method based on second-order attention, characterized in that, The method includes: S1. Acquire and process initial image features. Use the second-order attention multimodal ViT mutual enhancement fusion module as the backbone of the multimodal tracker to perform feature enhancement and fusion operations on the initial image features. Based on the attention weight-oriented fusion mechanism, integrate the original attention weights of different modal branches and perform another attention processing to enhance the original attention distribution, so as to embed the modal fusion process into the modal feature extraction process. S1 includes: S11. Given RGB and TIR template image pairs and RGB and TIR search region image pairs, divide the images in the RGB and TIR template image pairs and the RGB and TIR search region image pairs into image blocks; thereby obtaining a block embedding layer to process the image sequence and obtain the initial image features of the RGB and TIR template image pairs and the RGB and TIR search region image pairs. S12. The initial image features are processed in series to obtain image series features. The image series features are input into the backbone network for forward propagation. The backbone network includes: an L-layer transformer block and an acaformer block. S13. Perform a general attention process in the transformer to extract specific modal features; S14. In the acaformer block, the attention weights generated by the general attention process in different branches are treated as weight features using the attention mutual enhancement module AIE, thereby mining the consistency data of the attention weights in different modal branches. A second attention operation is performed on the original attention weights to enhance the original attention distribution, thereby obtaining the attention weights that integrate multiple modalities. The attention mutual enhancement module AIE is used to process the current adjustment region and its horizontally adjacent regions in the attention map to obtain attention mutual enhancement processing data, thereby obtaining fusion features. S15. Divide the features output by the backbone network into search region features and template features, concatenate the search region features of the two modalities, input them into the tracking head, and obtain the final tracking result. S2. Using the unified candidate elimination module, all features are sorted by integrating multimodal attention weights, and subsequent features are eliminated according to a predetermined ratio, so as to retain the preceding features.

2. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S11, a learnable position code is added. and The prior location information is processed using the following logic: In the formula, the subscript r represents RGB, t represents TIR, z represents the template, and x represents the search area.

3. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S12, the forward propagation operation is performed on the concatenated features of the image using the following logic to obtain the multimodal tracker: In the formula, The output of the last transformer block, where a box head consistent with the tracking head in OSTrack is used, is represented as .

4. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S14, the attention mutual enhancement module AIE is embedded in the second-order attention multimodal ViT mutual enhancement fusion module ACA to perform a joint optimization operation on the attention weights of different modal branches.

5. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S14, the attention map is generated from the RGB features using the following logic. : In the formula, within the Transformer's attention mechanism, three linear layers project the input to obtain query, key, and value, which correspond to the parameters respectively: , , .

6. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S14, the regions in the attention map are processed using the following logic. , wherein the region This characterizes the association between the template features and the search region features: In the formula, This indicates LayerNorm. It is a learnable linear projection. and Considering the presence of the softmax operation, the horizontally adjacent regions in the attention graph influence each other. Therefore, the horizontally adjacent regions in the attention graph are defined here. and All are used as input for AIE; Among them, let , ,have: In the formula, the subscript ' indicates that it is a variable in the AIE calculation.

7. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, In step S2, consistent weights are assigned to the corresponding positions of images in different modalities.

8. The multimodal adaptive fusion method based on second-order attention according to claim 1, characterized in that, S2 includes: S21. Given a query vector Here, the two query vectors are the query vectors corresponding to the template center tokens of the two modalities; S22. Select the query vector corresponding to the token at the center of the template and assign a scalar to the token in the RGB modal search region using the following logic. Token allocation for TIR modal search regions : In the formula, and It is the key vector of the search area token; S23, Using the first scalar Second scalar The tokens in the search area are sorted, and the preceding tokens are retained to filter out features with low target relevance.

9. A multimodal adaptive fusion system based on second-order attention, used to execute the multimodal adaptive fusion method based on second-order attention as described in any one of claims 1 to 8, characterized in that, The system includes: The second-order attention multimodal ViT mutual enhancement fusion module is used to collect and process initial image features. Using the second-order attention multimodal ViT mutual enhancement fusion module as the backbone of the multimodal tracker, feature fusion operation is performed on the initial image features. Based on the original attention weights, the original attention weights of different modal branches are combined according to the attention weight fusion mechanism, and then another attention processing is performed to enhance the original attention distribution, so as to embed the modality fusion process into the modality feature extraction process. The unified candidate elimination module is used to sort all features by integrating multimodal attention weights, filter out subsequent features according to a predetermined ratio, and retain preceding features accordingly.

Images