Efficient attention network implementation method and system based on rgb-t salient object detection

Abstract

The application discloses a kind of based on RGB-T salient target detection high-efficiency attention network implementation method and system, the method includes the following steps: S1, respectively extracts RGB image and thermal infrared image features by double-flow backbone network;S2, by the synergistic mechanism of channel attention and spatial attention, RGB image features and thermal infrared image features are carried out depth interaction and semantic alignment, generate fusion features;S3, by parallel dilated convolution, the multi-scale context of RGB branch, thermal infrared branch and cross-modal fusion branch is captured respectively, and each branch feature is weighted aggregated, and the multi-scale feature of optimization is output;S4, the multi-scale feature of optimization is adaptively flowed and aggregated, and aggregated feature is obtained;S5, based on aggregated feature, the salient target detection result of RGB branch, thermal infrared branch and cross-modal fusion branch is output;The application is realized high-precision cross-modal salient target detection by the adaptive feature interaction and aggregation mechanism of fusion multi-modal attention.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically to a method and system for implementing an efficient attention network based on RGB-T salient object detection. Background Technology

[0002] Satisfactory object detection (SOD), a fundamental and crucial task in computer vision, aims to simulate human visual attention mechanisms by automatically identifying and segmenting the most attractive or important regions in an image. As a core branch of computer vision, it has been widely applied in various downstream tasks, such as object detection, semantic segmentation, visual understanding, video analysis, and robotics perception and autonomous driving, demonstrating broad application value.

[0003] Single RGB modal imaging is limited in low-light, rain, and fog conditions, resulting in insufficient robustness and accuracy, and difficulty in providing complete target details. Thermal infrared (TIR) ​​images, on the other hand, are insensitive to illumination and can complement RGB images in complex scenes, providing additional saliency information and improving detection performance. In recent years, researchers have begun to focus on fusing TIR and RGB images, developing RGB-T salient target detection technology. Infrared images can capture the thermal radiation signals of object surfaces, are insensitive to changes in illumination, and can still clearly obtain target outlines and structures in low light and complex backgrounds. RGB and TIR images complement each other: RGB provides rich color and texture details, while TIR supplements object information in low-light and occluded conditions. Through multimodal fusion, the ability to identify target regions can be enhanced, compensating for the shortcomings of a single modality and improving the adaptability and accuracy of detection in diverse scenes.

[0004] Traditional RGB-T salient object detection methods mostly rely on hand-designed features for salient region extraction. These methods attempt to fuse low-level features from two modalities through graph learning, manifold ranking, or low-rank tensor models. For example, Wang et al. proposed multi-task manifold ranking to achieve collaborative integration through cross-modal consistency constraints. Huang et al. used multimodal low-rank tensor models to suppress redundant information and noise in features. Although these traditional methods have achieved certain results under specific conditions, they are highly dependent on hand-designed features, have limited generalization ability, and lack an understanding of high-level semantic information in images, making it difficult to accurately capture salient targets in complex scenes.

[0005] In recent years, the rapid development of deep learning technologies such as self-attention mechanisms based on the Transformer architecture has driven significant breakthroughs in the field of RGB-T salient object detection. For example, Liu et al. used a dual-Swin Transformer structure to extract hierarchical features and utilized a self-attention mechanism to capture cross-modal information. Liao et al. designed a multimodal interactive attention module to filter out interfering information; Zhang et al. proposed using the primary modality to generate prototypes to guide the semantic enhancement of auxiliary modalities. Furthermore, to address the potential weak alignment problem in RGB-T image pairs, Tu et al. proposed DCNet to model the correlation between modalities through dynamic convolution and spatial affine transformation, but this method suffers from high computational cost. However, existing methods still have significant shortcomings: multi-scale fusion lacks a selective mechanism and is computationally complex; interactive attention weights are computationally simple but lack cross-guidance; weak alignment processing is computationally expensive and may disrupt spatial consistency. These limitations restrict the robustness and efficiency of RGB-T SOD in complex scenarios. Summary of the Invention

[0006] The purpose of this invention is to provide an efficient attention network implementation method and system for RGB-T salient target detection. By integrating the adaptive feature interaction and aggregation mechanism of multimodal attention, high-precision cross-modal salient target detection can be achieved.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An efficient attention network implementation method based on RGB-T salient object detection includes the following steps: S1. Extract RGB image features and multi-scale features of thermal infrared images respectively through a dual-stream backbone network; S2. In the encoding stage, through the collaborative mechanism of channel attention and spatial attention, cross-modal deep interaction and semantic alignment are performed on the multi-scale features of RGB images and thermal infrared images layer by layer to generate fused features; S3. In the decoding stage, multi-scale context of the RGB branch, thermal infrared branch and cross-modal fusion branch are captured by parallel dilated convolution, and the features of each branch are weighted and aggregated based on the adaptive attention mechanism to output the optimized multi-scale features. S4. By using gated modulation and dynamic weighting strategies, the optimized multi-scale features are adaptively flowed and aggregated to obtain aggregated features; S5. Based on aggregated features, the significant target detection results of the RGB branch, thermal infrared branch and cross-modal fusion branch are output respectively through a multi-head supervision mechanism.

[0008] Furthermore, S2, through a collaborative mechanism of channel attention and spatial attention, involves a process of cross-modal deep interaction and semantic alignment of multi-scale features of RGB and thermal infrared images layer by layer, specifically as follows: S2.1 Dimensionality reduction of RGB and thermal infrared features using 1×1 convolution; S2.2 Calculate the channel attention weights after dimensionality reduction of multi-scale features of RGB and thermal infrared images, and perform cross-modal cross-guided enhancement and stitching; Spatial attention weights for multi-scale features of RGB and thermal infrared images are calculated based on the stitching results.

[0009] Furthermore, the specific expression for dimensionality reduction of the multi-scale features of the RGB image and the thermal infrared image is as follows: (2) Among them, the multi-scale features of RGB images Multi-scale features of thermal infrared images R represents the multi-scale features of the RGB image after dimensionality reduction; T represents the multi-scale features of the thermal infrared image after dimensionality reduction. This represents a 1×1 convolution operation, which converts the input features from... Dimensional reduction to dimensionality dimension; The expression for the channel attention weight is as follows: (3) in, These are the channel attention weights after dimensionality reduction of the multi-scale features of the RGB image; These are the channel attention weights after dimensionality reduction of the multi-scale features of the thermal infrared image; This represents a combination of global average pooling and global max pooling operations. The Sigmoid activation function is used, and MLP is a multilayer perceptron. Then, cross-modal cross-guided enhancement is performed, with the expression: (4) in This indicates element-wise multiplication. To enhance the multi-scale features of the RGB image after cross-guided enhancement; Multiscale features of cross-guided enhanced thermal infrared images; Spatial attention weights The expression is: (5) in, Indicates feature splicing, Represents the convolution operation; The expression for performing cross-modal depth interaction and semantic alignment of multi-scale features of RGB and thermal infrared images layer by layer is as follows: (6) (7) in, This indicates a fusion convolution operation. A 1×1 convolution for residual connections; It is a unified characterization after cross-guidance, used to suppress noise introduced by modal differences and misalignment.

[0010] Furthermore, the optimization process of the multi-scale features in S3 is specifically as follows: Multi-branch features are processed in parallel using convolution operations with different dilation rates to obtain RGB branch, thermal infrared branch and cross-modal fusion branch; All branch outputs are concatenated, and adaptive attention weights for the RGB branch, thermal infrared branch, and cross-modal fusion branch are calculated using global average pooling and multilayer perceptron. The RGB branch, thermal infrared branch, and cross-modal fusion branch are weighted and aggregated based on adaptive attention weights to output optimized multi-scale features.

[0011] Furthermore, the process of parallel processing of multi-branch features using convolution operations with different dilation rates specifically involves: Given input features Let the set of expansion rates be... The i-th branch is defined as: (8) in Let be the expansion rate of the i-th branch. Indicates the expansion rate 3×3 convolution; The adaptive attention weights of the RGB branch, thermal infrared branch, and cross-modal fusion branch The expression is as follows: (9) in arrive This represents the features extracted by the four parallel convolutional branches under different receptive fields. Used to characterize the importance of the features extracted by the four parallel convolution branches.

[0012] Furthermore, the expression for weighted aggregation of the RGB branch, thermal infrared branch, and cross-modal fusion branch based on adaptive attention weights is as follows: (10) in This indicates the importance of the features extracted by the i-th convolutional branch; The expression for the optimized multi-scale feature is as follows: (11)

[0013] in, This is a residual connection.

[0014] Furthermore, S4, the process of adaptive flow and aggregation of the optimized multi-scale features, specifically involves: The modulation weights of the RGB branch and the thermal infrared branch are calculated using a gated network; The modulation features are obtained by multiplying the fused features element by element based on the modulation weights. The modulation weights of the RGB branch and the thermal infrared branch are concatenated with the fusion features, and the dynamic fusion weights are calculated through a weight learner. The modulation weights of the RGB branch and the thermal infrared branch are weighted and fused based on dynamic fusion weights, and aggregated features are output through feature enhancement network and residual connection.

[0015] Furthermore, the process of calculating the modulation weights of the RGB branch and the thermal infrared branch using a gating network is specifically as follows: First, given the fused feature F, RGB feature R, and thermal infrared feature T, the modulation weights are calculated using a gating network: (12) in and The gated networks for RGB and thermal infrared branches are respectively composed of two 1×1 convolutional layers; It is the modulation weight of the RGB branch; It is the modulation weight of the thermal infrared branch; The expression for the modulation feature is specifically as follows: (13) in, It is a modulation feature of the RGB branch; It is a modulation characteristic of the thermal infrared branch; The expression for the dynamic fusion weight is as follows: (14) in Dynamic weights for the RGB and thermal infrared branches.

[0016] Furthermore, the expression for weighted fusion of the modulation weights of the RGB branch and the thermal infrared branch based on dynamic fusion weights is specifically as follows: (15) in and The weights for the RGB and thermal infrared branches are respectively. The aggregation feature The specific expression is: (16) in, The feature enhancement network consists of two 3×3 convolutional layers.

[0017] The present invention also provides a system for implementing an efficient attention network method for RGB-T salient object detection, comprising: A dual-stream Res2Net backbone network is used to extract features from RGB image and thermal infrared image. The efficient attention fusion module is used in the encoding stage to perform cross-modal deep interaction and semantic alignment of RGB image features and thermal infrared image features layer by layer through the collaborative mechanism of channel attention and spatial attention, so as to generate fused features; The selective fusion module is used to capture the multi-scale context of the RGB branch, thermal infrared branch and cross-modal fusion branch respectively through parallel dilated convolution during the decoding stage, and to perform weighted aggregation of the features of each branch based on an adaptive attention mechanism to output optimized multi-scale features. The feature flow module is used to adaptively flow and aggregate optimized multi-scale features through gating modulation and dynamic weighting strategies to obtain aggregated features. It is also used to output significant target detection results for RGB branch, thermal infrared branch and cross-modal fusion branch respectively based on aggregated features through a multi-head supervision mechanism.

[0018] According to specific embodiments provided by the present invention, the present invention has the following technical effects compared to the prior art: This invention employs a dual-stream backbone network to independently extract initial features from each modality, preserving the integrity of the original information for subsequent processing. During the encoding stage, a collaborative mechanism of channel and spatial attention guides the network layer by layer to focus on complementary and aligned semantic information between the two modalities, achieving deep cross-modal interaction and effectively overcoming modal differences to generate richer and more accurate fused features. In the decoding stage, a parallel dilated convolution strategy captures multi-scale contextual information from each branch, enhancing the model's adaptability to targets of different sizes. Meanwhile, an adaptive attention mechanism intelligently evaluates and weights the contributions of each branch, ensuring the selection and integration of optimal features and outputting optimized multi-scale features. Subsequently, gating modulation and dynamic weighting strategies further adaptively flow and aggregate these features, achieving refined information control and global optimization, ultimately resulting in highly cohesive and discriminative aggregated features. Through a multi-head supervision mechanism, independent supervised learning is performed on the three branches, which not only promotes the learning of features of each modality branch, but also enhances the performance of the fusion branch. This allows the model to fully leverage the synergistic effect of cross-modal fusion while maintaining the unique advantages of each modality, ultimately enabling it to output more accurate, complete, and robust salient target detection results in complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] The following description, in conjunction with the accompanying drawings, further illustrates the efficient attention network implementation method and system for RGB-T salient target detection of the present invention; Figure 1 This is an overall framework diagram of the efficient attention network implementation system based on RGB-T salient target detection in Embodiment 2 of the present invention; Figure 2 This is a structural diagram of the high-efficiency attention fusion module in Embodiment 2 of the present invention; Figure 3 This is a structural diagram of the selective fusion module in Embodiment 2 of the present invention; Figure 4 This is a structural diagram of the feature flow module in Embodiment 2 of the present invention; Figure 5 These are visual comparison images of different network models in Embodiment 3 of the present invention; Figure 6 This is a visual comparison of the ablation experiment in Embodiment 3 of the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0022] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0023] Example 1 This invention provides a method for implementing an efficient attention network based on RGB-T salient object detection, comprising the following steps: S1. Extract RGB image features and thermal infrared image features respectively through a dual-stream backbone network; This embodiment specifically involves: First, extracting multi-scale feature representations from RGB and thermal infrared images using a dual-stream Res2Net backbone. Then, during the encoding stage, an EAM module is introduced layer by layer to achieve deep interaction and semantic alignment of cross-modal features. In the decoding stage, the RGB and thermal infrared branches respectively employ an SFM module for multi-scale context aggregation, while the fusion branch uses SFM for progressive upsampling and interacts with the single-modal branch through FFM. Finally, a multi-head supervision mechanism outputs the prediction results of the three branches. The overall algorithm architecture is as follows: Figure 1 As shown, let the input be an RGB image. With thermal infrared images Feature extraction is performed using a dual-stream Res2Net backbone: (1).

[0024] S2. In the encoding stage, through the collaborative mechanism of channel attention and spatial attention, cross-modal deep interaction and semantic alignment are performed on RGB image features and thermal infrared image features layer by layer to generate fused features; This embodiment is specifically as follows: Figure 2 As shown, firstly, RGB features are given. and thermal infrared characteristics Feature dimensionality reduction is performed using 1×1 convolution: (2) in This represents a 1×1 convolution operation, which converts the input features from... Dimensional reduction to dimensionality dimension.

[0025] Cross-guided attention is a core innovation of EAM, achieving cross-modal feature enhancement through the synergistic effect of channel attention and spatial attention. For RGB and thermal infrared features, channel attention weights are calculated separately: (3) in This represents a combination of global average pooling and global max pooling operations. The activation function is Sigmoid, and the MLP is a multilayer perceptron. Then, cross-modal channel attention is used for feature guidance. (4) in This indicates element-wise multiplication. This cross-guided mechanism enables RGB features to be enhanced based on the attention distribution of the thermal infrared modality, achieving deep cross-modal interaction.

[0026] Next, the cross-guided features are concatenated, and spatial attention weights are calculated: (5) in Indicates feature splicing, This represents the convolution operation.

[0027] The final feature fusion is achieved through spatial attention enhancement and residual connections: (6) (7) in This indicates a fusion convolution operation. It is a 1×1 convolution for residual connections. It is a unified characterization after cross-guidance, which successfully suppresses noise introduced by modal differences and misalignment.

[0028] S3. In the decoding stage, multi-scale context of the RGB branch, thermal infrared branch and cross-modal fusion branch are captured by parallel dilated convolution, and the features of each branch are weighted and aggregated based on the adaptive attention mechanism to output the optimized multi-scale features. This embodiment is specifically as follows: Figure 3 As shown, SFM employs four parallel branches to process contextual information at different scales, given input features. Let the set of expansion rates be... The i-th branch is defined as: (8) in Let be the expansion rate of the i-th branch. Indicates the expansion rate The 3×3 convolutions allow each branch to capture contextual information from different receptive fields.

[0029] Then, after concatenating the outputs of all branches, attention weights are calculated using global pooling and a multilayer perceptron. (9) in arrive This represents the features extracted by the four parallel convolutional branches under different receptive fields. Used to characterize the importance of the features extracted by the four parallel convolution branches.

[0030] Next, the features of each branch are weighted and fused based on attention weights: (10) in This indicates the importance of the features extracted by the i-th parallel convolutional branch. This selective mechanism can automatically learn the importance of context at different scales and suppress redundant responses.

[0031] Finally, the final output is obtained by fusing convolutions and residual connections: (11)

[0032] in For residual connections, SFM utilizes 1×1 convolutions to solve the dimension matching problem between input and output channels, thereby enabling explicit learning and dynamic weight allocation of multi-scale features. This design allows the module to adaptively optimize feature fusion, ultimately achieving the effects of suppressing redundant responses and enhancing feature discriminability and representational capabilities.

[0033] S4. By using gated modulation and dynamic weighting strategies, the optimized multi-scale features are adaptively flowed and aggregated to obtain aggregated features; In this specific embodiment, the following steps are taken: First, given the fused feature F, the RGB feature R, and the thermal infrared feature T, the modulation weights are calculated using a gating network: (12) in and The network consists of two 1×1 convolutional layers, representing RGB and thermal infrared branches respectively. The gated network generates corresponding modulation weights based on the feature distribution of each single-modality branch. These gate weights are then used to modulate the fused features. (13) By forcing branches with different expansion rates to compete, this module is endowed with the ability to adaptively adjust its receptive field based on the input content. It possesses receptive field adaptation capabilities, dynamically weighing the importance of global contextual information against local detailed features based on the semantic complexity of the input content. The modulated features are then concatenated with the original fused features, and dynamic weights are calculated using a weight learner. (14) in The dynamic weights are assigned to the RGB and thermal infrared branches. Then, the modulation features are weighted and fused based on these dynamic weights: (15) in and These represent the weights for the RGB and thermal infrared branches, respectively. The final output is obtained through a feature enhancement network and residual connections. (16) in The feature enhancement network consists of two 3×3 convolutional layers.

[0034] S5. Based on aggregated features, the significant target detection results of the RGB branch, thermal infrared branch and cross-modal fusion branch are output respectively through a multi-head supervision mechanism.

[0035] Example 2 like Figure 1 The present invention provides a system for implementing an efficient attention network method for RGB-T salient object detection, comprising: A dual-stream Res2Net backbone network is used to extract multi-scale features from RGB and thermal infrared images; In this embodiment, the overall network architecture of the High Efficiency Attention Network (EANet) consists of four parts: a dual-stream feature extraction module, a cross-modal fusion module, a multi-scale decoding module, and a dynamic aggregation module. It adopts a U-shaped encoding / decoding structure, using Res2Net as the dual-stream backbone network. The Efficient Attention Fusion Module (EAM) enables cross-modal feature fusion during the encoding stage, the Selective Fusion Module (SFM) performs multi-scale semantic extraction, and the Feature Stream Module (FFM) is introduced to achieve dynamic feature aggregation during the decoding stage. The efficient attention fusion module is used to achieve deep interaction and semantic alignment of cross-modal features through the collaborative mechanism of channel attention, cross-guided attention and spatial attention; Specifically, in practical applications, due to differences in equipment, shooting angles, lighting conditions, and other factors, RGB and TIR image features often exhibit misalignment, leading to inconsistencies between modal information. The core challenge lies in achieving the most effective feature alignment and fusion with minimal computational cost. To address this issue, this embodiment designs an efficient attention fusion module, which achieves deep interaction and semantic alignment of cross-modal features through a cross-guided attention mechanism.

[0036] The selective fusion module is used to perform multi-scale contextual filtering and aggregation on aligned features through a parallel multi-branch structure combined with an adaptive attention mechanism; Specifically, this embodiment addresses the issue that traditional multi-scale feature fusion methods lack a selective mechanism for contextual information at different scales, easily introducing redundant responses and affecting the discriminative power of feature representations. This embodiment designs a parallel multi-branch structure to flexibly model contextual information at different scales and introduces an advanced attention mechanism to achieve adaptive weighted aggregation. SFM, through explicit learning and dynamic adjustment of the weights of features at each scale, can automatically optimize the information fusion process based on different features of the input image, thereby effectively suppressing redundant responses and improving the discriminative power and representational ability of features.

[0037] The feature flow module is used to adaptively filter and fuse aggregated multi-scale features through gated modulation and dynamic weighting guided by single-modal features, so as to achieve adaptive flow and aggregation of cross-branch features. Specifically, in this embodiment, the information interaction between the fusion branch and the RGB and thermal infrared single-modal branches is crucial during the decoding stage, aiming to converge the complementary feature representations captured by each branch. However, most current methods, including architectures based on U-Net and its variants, generally employ simple concatenation or element-wise addition strategies for feature aggregation. This undifferentiated integration approach has significant limitations: it fails to effectively identify discriminative details (such as precise object edges) and background redundancy (such as irrelevant textures and noise) in shallow features. When these unfiltered shallow features are directly introduced into the decoding path, they may inevitably dilute the purity of high-level abstract semantic information, leading to typical degradation phenomena such as artifacts and blurred boundaries in the final predicted image. To address this issue, this embodiment designs a feature flow module, which achieves adaptive feature flow across branches and modalities through gated modulation and dynamic weight learning.

[0038] Based on the final features of the fusion branch, a multi-head supervision mechanism is used to output significant target detection results for RGB, thermal infrared, and fusion branches.

[0039] Example 3 This embodiment provides an experimental process for verifying the efficient attention network implementation system based on RGB-T salient target detection in Embodiment 2: The system in Example 2 is implemented based on the PyTorch framework, and all experiments were conducted on a hardware environment equipped with an NVIDIA A100 GPU and CUDA version 12.8. The input image size was uniformly adjusted to 352×352 during both the training and testing phases. The Adam optimizer was used during training, with an initial learning rate set to 5×10⁻⁶. -5 The learning rate scheduling uses a cosine annealing strategy, where T_max is 100 and the minimum learning rate is 1×10. -6 Without setting weight decay, the batch size was set to 24, the training period was 200 epochs, the gradient clipping threshold was set to 0.5, and the entire training process took about 18 hours.

[0040] To evaluate the effectiveness of the proposed method, experiments were conducted on six datasets, including three aligned datasets and three weakly aligned datasets: VT821, VT1000, and VT5000. VT821 contains 821 aligned image pairs, with noise added to some images to increase the challenge of the dataset; VT1000 contains 1000 visible light-thermal infrared aligned image pairs acquired in relatively simple scenes; and VT5000 consists of 5000 aligned image pairs, covering various target scales and complex scenes, and has been divided into training and test sets.

[0041] (1) Comparative experiments between different networks The experimental comparative analysis in Example 2 compares the proposed model with current mainstream salient target detection methods. The comparison methods include six methods: CSRNet, CGFNet, OSRNet, DCNet, LSNet, and CAVER. Evaluation metrics: To verify the effectiveness of the model in Example 2, four metrics were selected to evaluate model performance: E-measure (E), S-measure (S), weighted F-measure (F), and mean absolute error (MAE).

[0042] Table 1 presents the quantitative evaluation results of the proposed model and six other methods on three publicly aligned datasets for four evaluation metrics. "↑" indicates a higher metric is better, and "↓" indicates a lower metric is better. Specifically, on the VT821 dataset, except for S which is second, all other metrics are first. On the VT1000 and VT5000 datasets, E and F are first, and all other metrics are in the top two.

[0043] Figure 5 The presentation showcases representative visual comparison results, depicting various scenarios including large objects, small objects, multiple objects, and complex backgrounds. The first row shows a scene with multiple objects; the proposed method accurately detects all objects, while other methods perform poorly in identifying multiple salient targets. The second and third rows show scenes containing both large and small objects; other methods, besides the proposed method, perform poorly in recognizing objects of different sizes. The fourth and sixth rows demonstrate complex scenes where background noise significantly impacts saliency maps generated by other methods. However, the proposed method effectively suppresses background noise, generating reliable saliency maps with clear edges, intact structure, and stability.

[0044] Table 1 Comparison of accuracy of different network models

[0045] Table 2 Comparison of accuracy of different network models

[0046] (2) Comparison of network model parameters To quantitatively evaluate the model efficiency of EANet proposed in this invention, this embodiment evaluates it from two dimensions: computational complexity (FLOPs) and parameter count (Params). A comparative analysis is performed with current mainstream RGB-T SOD methods. As shown in Table 1, EANet (OSRNet) requires only 39.82 GFLOPs and 30.88 M parameters, significantly lower than methods such as ADF, MIDD, and CGFNet. Experiments demonstrate that EANet provides a feasible solution for real-time RGB-T saliency target detection in resource-constrained environments.

[0047] (3) Ablation test Ablation test results as follows Figure 6 As shown in Table 3, the effectiveness of each module is verified. As the quantitative results in Table 3 show, using the single-stream U-Net architecture with direct feature addition layer by layer as the baseline, removing any module leads to a significant decrease in the metrics of the three datasets, demonstrating the contribution of each module.

[0048] like Figure 6 As shown, in rows 1, 3, 4, and 6, removing certain modules significantly weakens the ability to remove background noise, hindering the identification of salient targets. In row 2, the quality of the results generated by the method after removing modules decreases, but the model in this invention can still accurately identify targets. In row 5, the saliency maps generated by other methods are relatively coarse and fail to accurately detect salient targets.

[0049] Table 3 shows the ablation experiments on the VT821 dataset.

[0050] In summary, the method and system of this invention address key issues in RGB-T salient object detection, such as inefficient cross-modal fusion, lack of multi-scale context, and high computational cost, by proposing an efficient attention network (EANet). This network works collaboratively through three core modules: the Efficient Attention Fusion Module (EAM) utilizes channel and spatial attention mechanisms to achieve cross-modal deep interaction and semantic alignment, effectively mitigating modal differences and misalignment issues; the Selective Fusion Module (SFM) employs a parallel multi-branch structure and adaptive attention to dynamically filter and fuse multi-scale contextual information, enhancing feature discriminative power; and the Feature Flow Module (FFM) introduces gated modulation and dynamic weighting strategies to optimize the adaptive flow and aggregation of cross-branch features during the decoding stage. Experimental results on multiple public datasets demonstrate that EANet outperforms existing mainstream methods in key metrics such as S-measure, weighted F-measure, and MAE, exhibiting excellent robustness and generalization ability while maintaining low computational complexity. Future work will focus on further lightweight design of the model and exploring its application potential in more complex visual scenarios.

[0051] The above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for implementing an efficient attention network based on RGB-T salient object detection, characterized in that, Includes the following steps: S1. Extract RGB image features and multi-scale features of thermal infrared images respectively through a dual-stream backbone network; S2. In the encoding stage, through the collaborative mechanism of channel attention and spatial attention, cross-modal deep interaction and semantic alignment are performed on the multi-scale features of RGB images and thermal infrared images layer by layer to generate fused features; S2, through a collaborative mechanism of channel attention and spatial attention, is a process of cross-modal deep interaction and semantic alignment of multi-scale features of RGB images and thermal infrared images layer by layer, specifically as follows: S2.1 Dimensionality reduction of RGB and thermal infrared features using 1×1 convolution; S2.2 Calculate the channel attention weights after dimensionality reduction of multi-scale features of RGB and thermal infrared images, and perform cross-modal cross-guided enhancement and stitching; Spatial attention weights for multi-scale features of RGB and thermal infrared images are calculated based on the stitching results. The specific expression for dimensionality reduction of the multi-scale features of the RGB image and the thermal infrared image is as follows: （2） Among them, the multi-scale features of RGB images Multi-scale features of thermal infrared images R represents the multi-scale features of the RGB image after dimensionality reduction; T represents the multi-scale features of the thermal infrared image after dimensionality reduction. This represents a 1×1 convolution operation, which converts the input features from... Dimensional reduction to dimensionality dimension; The expression for the channel attention weight is as follows: （3） in, These are the channel attention weights after dimensionality reduction of the multi-scale features of the RGB image; These are the channel attention weights after dimensionality reduction of the multi-scale features of the thermal infrared image; This represents a combination of global average pooling and global max pooling operations. The Sigmoid activation function is used, and MLP is a multilayer perceptron. Then, cross-modal cross-guided enhancement is performed, with the expression: （4） in This indicates element-wise multiplication. To enhance the multi-scale features of the RGB image after cross-guided enhancement; Multiscale features of cross-guided enhanced thermal infrared images; Spatial attention weights The expression is: （5） in, Indicates feature splicing, Represents the convolution operation; The expression for performing cross-modal depth interaction and semantic alignment of multi-scale features of RGB and thermal infrared images layer by layer is as follows: （6） （7） in, This indicates a fusion convolution operation. A 1×1 convolution for residual connections; It is a unified characterization after cross-guidance, used to suppress noise introduced by modal differences and misalignment; S3. In the decoding stage, multi-scale context of the RGB branch, thermal infrared branch and cross-modal fusion branch are captured by parallel dilated convolution, and the features of each branch are weighted and aggregated based on the adaptive attention mechanism to output the optimized multi-scale features. S4. By using gated modulation and dynamic weighting strategies, the optimized multi-scale features are adaptively flowed and aggregated to obtain aggregated features; S5. Based on aggregated features, the significant target detection results of the RGB branch, thermal infrared branch and cross-modal fusion branch are output respectively through a multi-head supervision mechanism.

2. The efficient attention network implementation method for RGB-T salient target detection according to claim 1, characterized in that, The optimization process for the multi-scale features in S3 is as follows: Multi-branch features are processed in parallel using convolution operations with different dilation rates to obtain RGB branch, thermal infrared branch and cross-modal fusion branch; All branch outputs are concatenated, and adaptive attention weights for the RGB branch, thermal infrared branch, and cross-modal fusion branch are calculated using global average pooling and multilayer perceptron. The RGB branch, thermal infrared branch, and cross-modal fusion branch are weighted and aggregated based on adaptive attention weights to output optimized multi-scale features.

3. The efficient attention network implementation method for RGB-T salient target detection according to claim 2, characterized in that, The process of parallel processing of multi-branch features using convolution operations with different dilation rates is specifically as follows: Given input features Let the set of expansion rates be... The i-th branch is defined as: （8） in Let be the expansion rate of the i-th branch. Indicates the expansion rate 3×3 convolution; The adaptive attention weights of the RGB branch, thermal infrared branch, and cross-modal fusion branch The expression is as follows: （9） in arrive This represents the features extracted by the four parallel convolutional branches under different receptive fields. Used to characterize the importance of the features extracted by the four parallel convolution branches.

4. The efficient attention network implementation method for RGB-T salient target detection according to claim 3, characterized in that, The expression for weighted aggregation of the RGB branch, thermal infrared branch, and cross-modal fusion branch based on adaptive attention weights is as follows: （10） in This indicates the importance of the features extracted by the i-th convolutional branch; The expression for the optimized multi-scale feature is as follows: （11） in, This is a residual connection.

5. The efficient attention network implementation method for RGB-T salient target detection according to claim 1, characterized in that, S4, the process of adaptive flow and aggregation of the optimized multi-scale features, specifically involves: The modulation weights of the RGB branch and the thermal infrared branch are calculated using a gated network; The modulation features are obtained by multiplying the fused features element by element based on the modulation weights. The modulation weights of the RGB branch and the thermal infrared branch are concatenated with the fusion features, and the dynamic fusion weights are calculated through a weight learner. The modulation weights of the RGB branch and the thermal infrared branch are weighted and fused based on dynamic fusion weights, and aggregated features are output through feature enhancement network and residual connection.

6. The efficient attention network implementation method for RGB-T salient target detection according to claim 5, characterized in that, The process of calculating the modulation weights of the RGB branch and the thermal infrared branch using a gated network is as follows: First, given the fused feature F, RGB feature R, and thermal infrared feature T, the modulation weights are calculated using a gating network: （12） in and The gated networks for RGB and thermal infrared branches are respectively composed of two 1×1 convolutional layers; It is the modulation weight of the RGB branch; It is the modulation weight of the thermal infrared branch; The expression for the modulation feature is specifically as follows: （13） in, It is the modulation characteristic of the RGB branch; It is a modulation characteristic of the thermal infrared branch; The expression for the dynamic fusion weight is as follows: （14） in Dynamic weights for the RGB and thermal infrared branches.

7. The efficient attention network implementation method for RGB-T salient target detection according to claim 5, characterized in that, The specific expression for weighted fusion of the modulation weights of the RGB branch and the thermal infrared branch based on dynamic fusion weights is as follows: （15） in and The weights for the RGB and thermal infrared branches are respectively. The aggregation feature The specific expression is: （16） in, The feature enhancement network consists of two 3×3 convolutional layers.

8. A system for implementing an efficient attention network based on RGB-T salient object detection, used to execute the efficient attention network implementation method for RGB-T salient object detection as described in any one of claims 1-7, characterized in that, include: A dual-stream Res2Net backbone network is used to extract features from RGB image and thermal infrared image. The efficient attention fusion module is used in the encoding stage to perform cross-modal deep interaction and semantic alignment of RGB image features and thermal infrared image features layer by layer through the collaborative mechanism of channel attention and spatial attention, so as to generate fused features; The selective fusion module is used to capture the multi-scale context of the RGB branch, thermal infrared branch and cross-modal fusion branch respectively through parallel dilated convolution during the decoding stage, and to perform weighted aggregation of the features of each branch based on an adaptive attention mechanism to output optimized multi-scale features. The feature flow module is used to adaptively flow and aggregate optimized multi-scale features through gated modulation and dynamic weighting strategies to obtain aggregated features; It is also used to output significant target detection results for RGB branch, thermal infrared branch and cross-modal fusion branch respectively based on aggregated features and through a multi-head supervision mechanism.

Images