Lightweight rgb-t salient object detection method

Abstract

The application relates to the technical field of computer vision, and discloses a light-weight RGB-T salient object detection method, a local-global complementary aggregation module is designed at a feature interaction level, local detail enhancement is completed through shallow convolution and bilinear upsampling, global semantic modeling is realized by using a global average pooling and a full connection layer, operation cost and memory occupation are reduced, and the consistency and discriminativeness of cross-level information are maintained; a triple clue mining module is introduced at a feature mining level, low-cost operators such as multi-scale dilated convolution, point-by-point convolution and discrete cosine transformation are used to model spatial context, channel dependency and frequency semantics, the capture of cross-modal details and complementary information is ensured, and the parameter quantity and the calculation complexity are reduced; at a training optimization level, channel consistency loss and deep saliency supervision loss are used to effectively compensate for the feature alignment deficiency and the shallow expression weakening problem caused by the limited network capacity of the light-weight network, and the model detection capability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a lightweight RGB-T salient object detection method. BACKGROUND

[0002] Salient object detection (SOD) is an important research direction in the field of computer vision, which aims to simulate the human visual attention mechanism, and automatically highlight and detect the most salient object region from the input image through automatic means. With the development of artificial intelligence and deep learning, salient object detection has wide application value in intelligent monitoring, automatic driving, unmanned aerial vehicle inspection, medical image analysis and other scenes. Early salient object detection methods mainly rely on low-level features and prior rules, such as color, texture, contrast and spatial distribution feature calculation, combined with center prior, background prior and other heuristic methods. This kind of method can achieve certain effect in simple environment, but in complex background, low illumination, occlusion and camouflage and other real application conditions, its robustness and generalization ability are seriously insufficient, which leads to poor detection accuracy of early salient object detection methods in real application conditions.

[0003] The rise of deep learning has promoted salient object detection methods based on convolutional neural networks, which can extract multi-scale semantic information end-to-end and significantly improve detection accuracy and stability using attention mechanism and feature fusion strategy. However, relying only on a single RGB modality still has performance degradation problems in insufficient light, cluttered background or cross-scene conditions. To solve this problem, multi-modal salient object detection has gradually attracted attention. RGB-D methods supplement spatial geometric information through depth maps, while RGB-T methods fuse visible light and thermal infrared modalities, and use their complementary characteristics to show higher robustness in night, low light and complex environments. At the same time, with the development of artificial intelligence from the cloud to the terminal and edge side, salient object detection also faces new development trends: how to realize lightweight and high efficiency on edge devices. Traditional deep models often have large number of parameters and complex calculations, which are difficult to realize real-time inference on mobile terminals, embedded AI chips, unmanned aerial vehicles and Internet of Things devices. Therefore, combining lightweight network design and cross-modal efficient fusion mechanism to develop an RGB-T salient object detection method with high precision and low power platform demand has become an important research direction and urgent need in this field.

[0004] In the RGB-based salient object detection method, J. Zhao et al. proposed a salient object detection method based on optical remote sensing image, which adopted recursive adaptive graph reasoning network (RAGRN) and region and boundary interaction mechanism. This method effectively enhanced the fusion of target region and boundary information, and improved the target detection ability in complex remote sensing images. Z. Wu et al. proposed a cascaded partial decoder structure (Cascaded Partial Decoder), which processed features in stages, not only accelerated the calculation process of salient object detection, but also improved the detection accuracy, especially suitable for real-time processing application scenarios. The existing RGB-based salient object detection method can achieve good performance under ideal lighting and clear background conditions, but in real environments such as low light, complex background and target occlusion, the detection results are often unstable and difficult to meet the needs of complex application scenarios.

[0005] To improve the stability of salient object detection results, researchers have proposed RGB-T salient object detection methods to improve detection accuracy by fusing the complementary information of visible light and thermal infrared modalities. In the RGB-T salient object detection method, Q. Zhang et al. proposed a method for RGB-T salient object detection by fusing multi-layer convolutional neural network (CNN) features; this method effectively improves the salient object detection accuracy in RGB-T images by fusing multi-scale features of RGB and infrared (T) images. Z. Tu et al. proposed an RGB-T salient object detection method based on collaborative graph learning; by constructing a collaborative graph, the feature relationship between RGB and infrared modalities is dynamically adjusted, significantly enhancing the target detection ability in complex backgrounds. H. Zhou et al. proposed an RGB-T salient object detection method based on location-aware relationship learning; this method improves the accuracy and stability of target detection by learning the relationship between target location and multi-modal features, especially in dynamic environments. F. Huo et al. proposed an efficient context-guided stacked refinement network (CSRN), which effectively improves the accuracy and efficiency of RGB-T salient object detection by guiding network to fuse context information, especially in low light and complex environments. W. Zhou et al. proposed an effective and consistent feature fusion network (ECFFNet), which improves the accuracy of RGB-T salient object detection by optimizing the feature fusion strategy, especially in the application of cross-modal feature fusion and consistency loss function. Z. Tu et al. proposed a multi-interaction dual decoder (MIDD) model, which strengthens the collaborative effect between RGB and infrared image modalities through a dual decoder structure and multi-interaction mechanism, significantly improving detection accuracy.

[0006] Although the existing RGB-T salient object detection method has made progress in accuracy, most of the existing methods only focus on the feature interaction in the spatial domain or the channel domain, ignoring the complementary role of the frequency domain semantics, resulting in insufficient cross-modal feature alignment and fusion, making it difficult to fully play the complementary advantages of RGB and T modalities, and further unable to obtain more rich image features, and unable to achieve higher precision salient object detection. And most of the mainstream RGB-T detection models rely on large-scale backbone networks such as ResNet and VGG, and superimpose complex multi-layer feature fusion structures. Although this design improves the accuracy, it brings extremely high computational cost and storage overhead, and the efficiency of salient object detection is low, which is difficult to promote in real-time applications. In addition, with the AI application gradually moving from the cloud to the terminal and edge, the actual application scenarios (such as unmanned aerial vehicle inspection, intelligent security, vehicle-mounted equipment) have higher requirements for lightweight and real-time; However, the existing RGB-T salient object detection method has not been optimized in terms of parameter size, computational delay and energy consumption for edge devices, making it difficult to realize fast reasoning and efficient operation on embedded AI chips, mobile terminals or Internet of Things devices, and further unable to realize deployment in edge devices, limiting the application scenarios.

[0007] In summary, the existing technology has deficiencies in cross-modal feature fusion, which cannot effectively fuse the features extracted from RGB images and thermal infrared images, and is prone to lose key features during fusion, resulting in insufficient fusion, low target detection accuracy, and unstable detection results. To improve detection accuracy, the existing technology relies on large-scale backbone networks for feature fusion, which brings extremely high computational cost and high storage overhead, cannot realize lightweight deployment in edge devices, and has low detection efficiency for salient objects. SUMMARY

[0008] Therefore, the technical problem to be solved by the present application is to overcome the problem of low target detection accuracy due to insufficient multi-modal feature fusion caused by ignoring the frequency domain semantics in the prior art, and the problem of high computational cost and low detection efficiency caused by relying on large-scale backbone networks.

[0009] To solve the above technical problems, the present application provides a lightweight RGB-T salient object detection method, comprising:

[0010] An RGB image to be detected and its corresponding thermal infrared image are obtained, and are input into a visible light feature extraction branch and a thermal infrared feature extraction branch of an encoder of MobileNetV2 respectively, to obtain visible light features and thermal infrared features;

[0011] The first visible light feature and the first thermal infrared feature are spliced as the first A multimodal fusion feature; let the first The multimodal fusion features are sequentially processed through global average pooling and fully connected layers to obtain global context features;

[0012] Global context features, the first The first multimodal fusion feature and the second The first visible light feature / the first The first thermal infrared feature is input into the visible light feature optimization branch / the first thermal infrared feature optimization branch. In the local-global complementary aggregation module, long-range spatial features are extracted and compared with the first... The first visible light joint feature / the first The first thermal infrared joint feature is added together to obtain the second feature. The first visible light fusion feature / the first One thermal infrared fusion feature; ;

[0013] The first The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature;

[0014] The first The first visible light branching cue feature, the first By splicing together the features of the thermal infrared branch cue, the first... A multimodal fusion feature;

[0015] The first The visible light feature and the first After splicing the first thermal infrared feature, input the second The decoding unit obtains the nth decoding unit. One decoding unit;

[0016] For the Upsample the significant prediction result of the decoding unit to obtain the first... The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the nth decoding unit. Significant prediction results for each decoding unit;

[0017] The salient prediction result of the first decoding unit is obtained as the salient target detection result of the RGB image to be detected.

[0018] Preferably, the RGB image to be detected and its corresponding thermal infrared image are acquired and input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and One thermal infrared feature, including:

[0019] The RGB image to be detected will be obtained. and its corresponding thermal infrared image The first visible light feature is input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, and the first visible light feature output from the first convolutional sampling unit in each branch is obtained. With the first thermal infrared feature , represented as:

[0020] ; ;

[0021] The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , represented as:

[0022] ; ;

[0023] in, , This represents the total number of coding units in the visible light feature extraction branch / thermal infrared feature extraction branch of the encoder in MobileNetV2. This represents depthwise separable convolution. Indicates downsampling; visible light characteristics Thermal infrared characteristics , , and These represent the height, width, and channels of the RGB image to be detected, respectively.

[0024] Preferably, the local-global complementary aggregation module includes:

[0025] The local complementary aggregation unit takes global context features as input, passes through a self-attention mechanism, upsampling and convolution in sequence, and outputs long-range spatial features.

[0026] Global complementary aggregation unit, with the first The first multimodal fusion feature, the first The first visible light feature / the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature / the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature / the first A combined thermal infrared feature;

[0027] The fusion unit combines the input long-range spatial features, the first... The first visible light joint feature / the first After adding the combined thermal infrared features, the output is the first... The first visible light fusion feature / the first A thermal infrared fusion feature.

[0028] Preferably, let the first The multimodal fusion features are sequentially processed through global average pooling and fully connected layers to obtain global context features. Using these global context features as input, they are sequentially processed through a self-attention mechanism, upsampling, and convolution to output long-range spatial features, including:

[0029] For the Multimodal fusion features Perform global average pooling to obtain the global feature vector for each channel. ;

[0030] A fully connected layer is used to perform a non-linear transformation on the global feature vector of each channel to obtain the corresponding channel attention weights. ;

[0031] Using channel attention weights for the first The multimodal fusion features are weighted and adjusted to obtain a weighted feature map. ;

[0032] The weighted feature map is concatenated with the channel features of all channels to obtain the global context features. ;

[0033] Applying a self-attention mechanism to global context features to obtain long-range spatial features. ;

[0034] in, Indicates the first The first multimodal fusion feature Each channel is located in eigenvalues ​​at that location , Indicates the first The total number of channels in a multimodal fusion feature; and Indicates learnable parameters, In the weighted feature graph, the first... Each channel is located in eigenvalues ​​at that location express convolution, Indicates upsampling, This represents the self-attention mechanism.

[0035] Preferably, with the first The first multimodal fusion feature, the first The first visible light feature / the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature / the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature / the first The combined thermal infrared features include:

[0036] The first The visible light branching cue features and the first By splicing together the features of the thermal infrared branch cue, the first... Multimodal fusion features , ;

[0037] No. Visible light joint features , represented as:

[0038] ;

[0039] No. Joint thermal infrared features , represented as:

[0040] ;

[0041] in, Indicates the first A multimodal fusion feature; This indicates a splicing operation.

[0042] Preferably, the first The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature includes:

[0043] The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the spatial feature extraction branch of the triple clue mining module. After passing through the multi-scale fusion module, pooling, convolution and activation in sequence, they are residually connected with the output of the multi-scale fusion module to output spatial context features.

[0044] The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the channel feature extraction branch of the triple clue mining module, and after sequentially passing through pooling, two layers of convolution and activation, they are combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output channel dependency features;

[0045] The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the frequency feature extraction branch of the triple clue mining module. The input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed. After aggregation by a multilayer perceptron, the summation is combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output frequency cue features;

[0046] The first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The visible light branching cue features and the first One thermal infrared branching cue feature.

[0047] Preferably, the triple clue mining module is used to obtain the first... The first visible light fusion feature and the first When considering the spatial context features, channel dependency features, and frequency cue features of thermal infrared fusion features, they all include:

[0048] In the spatial feature extraction branch, the input features are sequentially processed by the multi-scale fusion module. Pooling ,convolution With activation Then, a residual connection is performed with the output of the multi-scale fusion module to output the corresponding spatial context features, represented as: ;

[0049] In the channel feature extraction branch, the input features are sequentially processed through pooling, two convolutional layers, and activation, and then combined with the first... The first visible light fusion feature / the first The thermal infrared fusion features are used for residual connection, and the corresponding channel dependency features are output, represented as follows: ;

[0050] In the frequency feature extraction branch, the input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and then summed. After aggregation by a multilayer perceptron, the summation is combined with the first sub-band. The first visible light fusion feature / the first The thermal infrared fusion features are residually concatenated to output the corresponding frequency cue features, represented as follows: ;

[0051] in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light fusion feature; hour, Indicates the first One thermal infrared fusion feature; To express summation, This represents the two-dimensional discrete cosine transform.

[0052] Preferably, the first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature / the The thermal infrared branching cue features are represented as follows:

[0053] ;

[0054] in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light branching cue feature; hour, Indicates the first One thermal infrared branching cue feature.

[0055] Preferably, for the first Upsample the significant prediction result of the decoding unit to obtain the first... The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the nth decoding unit. Significant prediction results for each decoding unit include:

[0056] The first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results ;

[0057] The first The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... Features of fusion clues ;

[0058] The first Features of fusion clues With the Upsampling results Enter the number together The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , represented as: ;

[0059] in, , This indicates the total number of decoding units in the decoder of MobileNetV2.

[0060] Preferably, a total loss function is constructed for joint training of the encoder, the local-global complementary aggregation module, the triple clue mining module, and the decoder; the total loss function is expressed as:

[0061] ;

[0062] Channel consistency loss ;

[0063] Deep significance monitoring loss ;

[0064] in, and They represent the first The visible light feature and the first One thermal infrared feature, Indicates the first Significant prediction results for each decoding unit, Indicates the first The actual tag corresponding to each decoding unit Indicates channel Channel attention weights, Indicates the first The total number of channels corresponding to each decoding unit and Indicates the first The visible light feature and the first The first thermal infrared feature Characteristics of each channel Denotes the square of the L2 norm. Represents the binary cross-entropy. This indicates intersection, union, and ratio.

[0065] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0066] The lightweight RGB-T salient object detection method described in this invention uses MobileNetV2 as the backbone of the RGB-T stream. MobileNetV2 reduces computational cost through depthwise separable convolutions and enhances feature representation and gradient flow by combining inverse residual structures and linear bottlenecks, achieving both lightweight and high efficiency while maintaining accuracy, enabling real-time detection in edge devices. Simultaneously, this invention introduces a local-global complementary aggregation module between the encoder and decoder. By upsampling and convolutionally performing multimodal fusion features, it adds them to features that have undergone self-attention and convolution to obtain joint features, achieving the fusion of local image texture and cross-modal details. Compared to multi-layer convolution stacking, this significantly reduces parameters and computational cost, improving salient object detection efficiency. Global average pooling and small-scale fully connected layers are used to extract contextual semantics, constructing global contextual features from which long-range spatial features are extracted, avoiding the high computational overhead of large-scale attention networks and improving salient object detection efficiency while maintaining detection accuracy. The joint features are then added to the long-range spatial features to obtain the corresponding fusion features for subsequent decoding. The local-global complementary aggregation module effectively integrates local texture, global semantics, and modality-aware features to enhance cross-layer information consistency and semantic alignment. Simultaneously, the module maintains modularity and pluggability, allowing for flexible embedding into lightweight backbone networks. This significantly reduces model complexity while enhancing the interaction between local and global features, ensuring cross-layer information consistency and discriminative feature extraction. Furthermore, it enables the invention to maintain efficient inference and real-time response capabilities on edge devices, further improving detection accuracy and real-time performance in application scenarios with edge devices, and enhancing the efficiency of salient target detection.

[0067] The lightweight RGB-T salient target detection method described in this invention introduces a triple clue mining module between the encoder and decoder to achieve complementary fusion of spatial, channel, and frequency metrics. In spatial feature extraction, multi-scale spatial information is extracted using convolutional kernels with different dilation rates. Compared to stacking large-scale convolutional layers, only a small number of parameters are needed to complete multi-scale feature representation, avoiding the high computational overhead of deep networks. This increases the feature extraction scale, improves feature extraction accuracy, and accelerates the detection speed of salient targets. In channel feature extraction, pointwise convolution and sigmoid activation are used instead of complex multi-head attention mechanisms to achieve adaptive adjustment of weights between channels. The number of parameters is extremely low, while maintaining strong feature selection capabilities, making it suitable for fast inference on edge devices and deployable on edge devices. In frequency feature extraction, two-dimensional discrete cosine transform is used to decompose features in the frequency domain. This can extract complementary high and low frequency information without adding extra convolution stacking. Moreover, the two-dimensional discrete cosine transform is a deterministic operator that does not require training parameters. The computational complexity is much lower than that of deep network structures based on Fourier or wavelet, effectively avoiding parameter inflation and computational redundancy caused by complex attention networks, achieving model lightweighting, and further improving the efficiency of salient target detection. The triple-clue mining module jointly explores salient cues in the spatial, channel, and frequency domains, thereby extracting intramodal details and cross-modal complementary information, improving the progress of image feature extraction, and ensuring the accuracy of salient target detection. At the same time, through modular design, it can be flexibly embedded into a lightweight backbone network, maintaining high-precision cross-modal feature fusion capabilities while keeping the number of parameters and computational complexity within the range that edge devices can bear, improving the efficiency of salient target detection, and ensuring the overall method's deployability and real-time performance on edge devices.

[0068] This application employs channel consistency loss and deep saliency supervision loss during model training, effectively enhancing the discriminative power and robustness of the lightweight model without increasing inference computational overhead. Channel consistency loss learns adaptive weights to align the channel features of RGB and thermal infrared modalities, constraining cross-modal features to maintain consistency across corresponding dimensions. This compensates for insufficient cross-modal information fusion caused by the limited capacity of the lightweight backbone network, enabling the model to extract highly discriminative cross-modal complementary features with low computational cost. Through this loss constraint, even with shallow convolutions or simplified structures, the texture information of the RGB modality and the contour information of the thermal infrared modality can be fully aligned, thereby improving detection accuracy and boundary integrity. Deep saliency supervision loss applies a supervisory signal to the output of the network's multi-layer decoder, enabling intermediate and shallow features to participate in the learning of salient targets. This multi-layer supervision significantly improves the sensitivity of shallow features to salient regions, avoiding the problem of insufficient shallow representation in lightweight networks. Simultaneously, the combined supervision of binary cross-entropy and cross-union ratio (CUN) further optimizes the detail representation of boundary regions while maintaining global mask accuracy. This not only improves the overall robustness of the model but also ensures the stability of small-scale networks in complex scenarios, further enhancing the accuracy of salient target detection in the trained model and improving the stability of salient target detection in different application scenarios. Attached Figure Description

[0069] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein:

[0070] Figure 1 This is a flowchart of the lightweight RGB-T salient target detection method provided by the present invention;

[0071] Figure 2 This is a schematic diagram of a local-global complementary aggregation module;

[0072] Figure 3 This is a schematic diagram of the triple clue mining module. Detailed Implementation

[0073] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0074] Reference Figure 1 As shown in the flowchart, the lightweight RGB-T salient target detection method provided by the present invention includes the following specific steps:

[0075] Acquire the RGB image to be detected and its corresponding thermal infrared image The features are input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and thermal infrared features ;

[0076] The first The visible light feature and the first The first thermal infrared feature is stitched together as the first... Multimodal fusion features Order No. Multimodal fusion features Global context features are obtained by sequentially passing through global average pooling and fully connected layers. ;

[0077] global context features , No. Multimodal fusion features With the Visible light characteristics / th thermal infrared features The first branch in the input visible light feature optimization branch / thermal infrared feature optimization branch In the local-global complementary aggregation module, long-range spatial features are extracted and compared with the first... The first visible light joint feature / the first The first thermal infrared joint feature is added together to obtain the second feature. Visible light fusion features / th One thermal infrared fusion feature ; ;

[0078] The first Visible light fusion features / th One thermal infrared fusion feature The input is processed by a triple clue mining module, which extracts spatial context features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. Channel dependency characteristics Frequency cue features Add them together to get the first one. Visible light branching clue features / th Features of thermal infrared branch clues ;

[0079] The first Visible light branching clue features , No. Features of thermal infrared branch clues splice together to obtain the first A multimodal fusion feature;

[0080] The first Visible light characteristics With the thermal infrared features After splicing, enter the first The decoding unit sequentially undergoes convolution, batch normalization, and activation functions to output the nth decoding unit. Significant prediction results for each decoding unit ;

[0081] For the Significant prediction results for each decoding unit Perform upsampling to obtain the first Upsampling results ; will the first Visible light branching clue features With the Features of thermal infrared branch clues Add them together to get the first one. Features of fusion clues , and the Upsampling results Enter the first The decoding unit obtains the nth decoding unit. Significant prediction results for each decoding unit ;

[0082] Obtain the significant prediction result of the first decoding unit. This serves as the result of salient target detection in the RGB image to be detected.

[0083] The local-global complementary aggregation module includes:

[0084] Local complementary aggregation units, with global context features The input is processed sequentially through a self-attention mechanism, upsampling, and convolution, outputting long-range spatial features. ;

[0085] Global complementary aggregation unit, with the first Multimodal fusion features , No. Visible light characteristics / th thermal infrared features As input, let the first... Visible light characteristics / th thermal infrared features After self-attention and convolution, compared with the first convolution after upsampling... Multimodal fusion features Add them together and output the first result. Visible light joint features / th Joint thermal infrared features ;

[0086] The fusion unit integrates the long-range spatial features of the input. , No. Visible light joint features / th Joint thermal infrared features After adding, output the first... Visible light fusion features / th One thermal infrared fusion feature .

[0087] Specifically, the visible light feature extraction branch and the thermal infrared feature extraction branch of the encoder based on MobileNetV2 obtain... Visible light characteristics and One thermal infrared feature, including:

[0088] The RGB image to be detected will be obtained. and its corresponding thermal infrared image The first visible light feature is input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, and the first visible light feature output from the first convolutional sampling unit in each branch is obtained. With the first thermal infrared feature , represented as:

[0089] ; ;

[0090] The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , represented as:

[0091] ; ;

[0092] in, , This represents the total number of coding units in the visible light feature extraction branch / thermal infrared feature extraction branch of the encoder in MobileNetV2. This represents depthwise separable convolution. Indicates downsampling; visible light characteristics Thermal infrared characteristics , , and These represent the height, width, and channels of the RGB image to be detected, respectively.

[0093] In this embodiment, feature stitching refers to connecting two features laterally along the channel dimension; feature addition refers to adding the pixels on the two feature maps corresponding to the two features element by element.

[0094] Reference Figure 2 The diagram shown is a schematic of a local-global complementary aggregation module. The local-global complementary aggregation module in this embodiment includes:

[0095] ① Obtain long-distance spatial features :

[0096] For the Multimodal fusion features Perform global average pooling to obtain the global feature vector for each channel. ;

[0097] A fully connected layer is used to perform a non-linear transformation on the global feature vector of each channel to obtain the corresponding channel attention weights. ;

[0098] Using channel attention weights for the first The multimodal fusion features are weighted and adjusted to obtain the weighted features. ;

[0099] The weighted features are concatenated with the channel features across all channels to obtain the global context features. ;

[0100] Applying a self-attention mechanism to global context features to obtain long-range spatial features. ;

[0101] in, Indicates the first The first multimodal fusion feature Each channel is located in eigenvalues ​​at that location , Indicates the first The total number of channels in a multimodal fusion feature; and Indicates learnable parameters, Indicating the weighted features, the first Each channel is located in eigenvalues ​​at that location express convolution, Indicates upsampling, This represents the self-attention mechanism;

[0102] ② Obtaining joint features of visible light Thermal infrared combined characteristics :

[0103] The first The visible light branching cue features and the first By splicing together the features of the thermal infrared branch cue, the first... Multimodal fusion features , ;

[0104] No. Visible light joint features , represented as:

[0105] ;

[0106] No. Joint thermal infrared features , represented as:

[0107] ;

[0108] in, Indicates the first A multimodal fusion feature; This indicates a splicing operation.

[0109] Reference Figure 3 The diagram shown is a schematic of the triple clue mining module; in this embodiment of the invention, the first... The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature includes:

[0110] The first Visible light fusion features / th One thermal infrared fusion feature The spatial feature extraction branch, fed into the triple clue mining module, passes sequentially through a multi-scale fusion module, pooling, convolution, and activation. It is then residually concatenated with the output of the multi-scale fusion module to output spatial context features. ;

[0111] The first Visible light fusion features / th One thermal infrared fusion feature The channel feature extraction branch fed into the triple clue mining module undergoes pooling, two convolutional layers, and activation sequentially, before being connected to the first... The first visible light fusion feature / the first Residual connections are performed on the thermal infrared fusion features to output channel dependency features. ;

[0112] The first Visible light fusion features / th One thermal infrared fusion feature The input features are fed into the frequency feature extraction branch of the triple clue mining module. After being decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed, they are aggregated by a multilayer perceptron and then compared with the first... The first visible light fusion feature / the first Residual connections are performed on the thermal infrared fusion features to output frequency cue features. ;

[0113] The first Visible light fusion features / th One thermal infrared fusion feature Spatial context features Channel dependency characteristics Frequency cue features Add them together to get the first one. Visible light branching clue features With the Features of thermal infrared branch clues , represented as:

[0114] ;

[0115] in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light branching cue feature; hour, Indicates the first One thermal infrared branching cue feature.

[0116] This embodiment utilizes a triple clue mining module to obtain the first... Visible light fusion features and the One thermal infrared fusion feature Spatial context features Channel dependency characteristics Frequency cue features At times, all include:

[0117] In the spatial feature extraction branch, the input features are sequentially processed by the multi-scale fusion module. Pooling ,convolution With activation Then, the corresponding spatial context features are output. , represented as: ;

[0118] In the channel feature extraction branch, the input features are sequentially processed through pooling, two convolutional layers, and activation, and the corresponding channel dependency features are output. , represented as: ;

[0119] In the frequency feature extraction branch, the input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and then summed. After aggregation by a multilayer perceptron, the corresponding frequency cue features are output. , represented as: ;

[0120] in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light fusion feature; hour, Indicates the first One thermal infrared fusion feature; To express summation, This represents the two-dimensional discrete cosine transform.

[0121] After introducing the triple feature mining module in this example, the third... Visible light branching clue features , No. Features of thermal infrared branch clues splice together to obtain the first Multimodal fusion features , with the first Multimodal fusion features , No. Visible light characteristics / th thermal infrared features As input, let the first... Visible light characteristics / th thermal infrared features After self-attention and convolution, compared with the first convolution after upsampling... Multimodal fusion features Add them together and output the first result. Visible light joint features / th Joint thermal infrared features ; will the first Visible light branching clue features With the Features of thermal infrared branch clues Add them together to get the first one. Features of fusion clues , and the Significant prediction results for each decoding unit upsampling results Enter the first The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , represented as: .

[0122] Among them, when At that time, the first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results .

[0123] Based on the above embodiments, in this embodiment of the invention, a total loss function is constructed to jointly train the encoder, the local-global complementary aggregation module, the triple clue mining module, and the decoder; the total loss function is expressed as:

[0124] ;

[0125] Channel consistency loss ;

[0126] Deep significance monitoring loss ;

[0127] in, and They represent the first The visible light feature and the first One thermal infrared feature, Indicates the first Significant prediction results for each decoding unit, Indicates the first The actual tag corresponding to each decoding unit Indicates channel Channel attention weights, Indicates the first The total number of channels corresponding to each decoding unit and Indicates the first The visible light feature and the first The first thermal infrared feature Characteristics of each channel Denotes the square of the L2 norm. Represents the binary cross-entropy. This indicates intersection, union, and ratio.

[0128] In summary, this invention proposes a lightweight RGB-T salient object detection method (SCF-Net) supporting edge devices. SCF-Net jointly explores spatial structure, channel dependence, and frequency semantics to achieve deep cross-modal feature fusion. Specifically, this invention designs a Triple Clue Mining (TCM) module based on Discrete Cosine Transform (DCT) to effectively integrate spatial structure, channel dependence, and frequency semantics, thereby extracting intra-modal details and cross-modal complementary information. Furthermore, this invention introduces a Local-Global Complementary Aggregation (LGCA) module to integrate local texture and global semantics, enhancing cross-layer semantic alignment and information consistency. With the help of a lightweight backbone network, SCF-Net contains only 8.39M parameters, supporting real-time inference.

[0129] Based on the above embodiments, in this embodiment, a local-global complementary aggregation module and a triple cue mining module are introduced simultaneously for salient target detection. The local-global complementary aggregation module effectively fuses local texture, global semantics, and modality-aware features to enhance cross-layer information consistency and semantic alignment. The triple cue mining module is used to jointly explore salient cues in the spatial, channel, and frequency domains, thereby enabling the extraction of intramodal details and cross-modal complementary information.

[0130] This invention uses MobileNetV2 as the backbone of the RGB-T flow. MobileNetV2 reduces computation through depthwise separable convolutions and improves feature representation and gradient flow by combining inverse residual structures with linear bottlenecks, achieving both lightweight and high efficiency while maintaining accuracy.

[0131] Reference Figure 1 As shown, the encoder has 5 encoding units, and each encoding unit contains depthwise separable convolutions. and downsampling operation Extracting visible light features and thermal infrared characteristics , represented as:

[0132] ;

[0133] ;

[0134] in, This indicates the height of the input RGB image. This indicates the width of the input RGB image. This represents the channels of the input RGB image. This indicates that the input is an RGB image. This indicates that a thermal infrared image is being input.

[0135] The Local-Global Complementary Aggregation (LGCA) module comprises Local Complementary Aggregation (LCA) and Global Complementary Aggregation (GCA), which together enhance fine-grained details and overall semantics. In terms of lightweight design, the LGCA module also fully considers the computational constraints of edge devices. LCA achieves the fusion of local textures and cross-modal details through shallow convolutions and upsampling operations, and aligns and enhances details of RGB and infrared features through upsampling and a small number of convolutions. Compared to traditional multi-layer deep convolutional structures, LCA only requires a single bilinear interpolation and a 3×3 convolution to complete local texture fusion, thus effectively reducing the number of convolutional kernels and computational overhead. It utilizes self-attention to refine fine-grained features within a small scope, avoiding the high complexity of global attention and significantly reducing memory usage and computational costs. GCA employs global average pooling to extract contextual information, and then combines it with a lightweight fully connected layer to complete channel weight learning. Compared with global modeling methods based on Transformer or multi-head attention, its parameter count and computational complexity are significantly reduced, avoiding the high computational overhead of large-scale attention networks. At the same time, global semantic information is applied to the feature map in a one-time weighted manner, eliminating the need to repeatedly calculate the attention matrix and ensuring inference speed on edge devices. Meanwhile, the overall structure of LGCA maintains modularity and pluggability, allowing for flexible embedding of lightweight backbone networks, significantly reducing model complexity while enhancing the interaction between local and global features. This design ensures cross-layer information consistency and discriminative feature extraction, enabling the invention to maintain efficient inference and real-time response capabilities on edge devices.

[0136] In the RGB branch, LCA first fuses the features. Upsampling with Matching resolution, then through self-attention Refine To enhance object perception and differentiation, joint features are obtained. :

[0137] ;

[0138] ;

[0139] in, This indicates a splicing operation. Indicates visible light branching cue characteristics, Indicates the characteristics of hot branching clues, the first The visible light branching cue features and thermal branching cue features of the layer are derived from the first layer. The TCM output of layer 5 yields the visible light branching cue features and thermal branching cue features of layer 5. and , express Convolution operation, This indicates 2× bilinear upsampling.

[0140] For GCA, global context features It effectively guides the weight adjustment of each channel in the feature map, and the global context From high-level features and China passed extract:

[0141] ;

[0142] in, First of all and The two feature maps are fused to obtain a new fused feature map. Next, the fused feature map Perform global average pooling to obtain the global feature vector for each channel. Then, a fully connected layer is used to process the global feature vector of each channel. Perform a nonlinear transformation to obtain the channel attention weights. Finally, the obtained channel attention weights are used. For fused feature maps After weighted adjustment, the weighted feature map is obtained. The generated weighted feature map As Operation output :

[0143] ; ; ;

[0144] ; ; ;

[0145] in, Represents the fused feature map The Middle Each channel is located in The value at that location, , For learnable parameters, For learnable parameters, Represents the weighted feature map The Middle The spatial location of each channel The eigenvalue at that location. For Applying self-attention to model long-range spatial dependencies yields long-range spatial features. .

[0146] LGCA output visible light fusion characteristics , represented as: .

[0147] Corresponding thermal fusion characteristics It was calculated in the same way.

[0148] In terms of lightweight design, the LGCA module consists of two parts: Local Complementary Aggregation (LCA) and Global Complementary Aggregation (GCA). Both of them ensure feature interaction capabilities while taking into account computational efficiency and parameter scale.

[0149] The Triple Feature Mining Module (TCM) of this invention enhances visible light fusion features by jointly utilizing spatial, channel, and frequency cues. Thermal fusion characteristics The TCM adaptively emphasizes structure, semantics, and texture to enhance saliency representation. In its lightweight design, the TCM not only achieves joint modeling of spatial, channel, and frequency cues but also optimizes its structure to address the computational limitations of edge devices. Specifically, the TCM employs low-overhead operators such as multi-scale convolution and discrete cosine transform (DCT) to effectively avoid parameter inflation and computational redundancy caused by complex attention networks. Furthermore, its modular design allows for flexible embedding into lightweight backbone networks (such as MobileNetV2) without relying on large-scale ResNet or Transformer structures. This mechanism maintains high-precision cross-modal feature fusion capabilities while keeping the number of parameters and computational complexity within the tolerance range of edge devices, thereby ensuring the real-time inference performance and stability of the model on resource-constrained platforms.

[0150] Spatial Context of TCM Extracted through a multi-scale fusion (MSF) module, and then... and convolution: MSF first applies convolutional kernels with different dilation rates to the input features to obtain multi-scale features. Then, after splicing the multi-scale features, use... Convolutional integration yields The final output multi-scale fusion features :

[0151] ; ;

[0152] ; ;

[0153] in, Indicates the expansion rate of convolution.

[0154] TCM channel dependency pass and two Convolutional capture, represented as:

[0155] ;

[0156] TCM frequency clues , The data is decomposed into four directional subbands {LL, LH, HL, HH} using a two-dimensional discrete cosine transform (DCT), and then... Aggregation, represented as:

[0157] ;

[0158] in, This represents the summation operation. For each channel, first calculate its two-dimensional DCT-II coefficients. Then let the vertical and horizontal cutoff frequency indices be respectively , Define four masks Four directional offspring are obtained by element-wise multiplying the channel coefficients with the four masks. :

[0159] ;

[0160] ;

[0161] ;

[0162] ;

[0163] ;

[0164] ;

[0165] ;

[0166] in, , , Definition and Similarly, for height, for width, for The number of channels, This indicates an element-wise dot product operation.

[0167] Ultimate visible light branching cue features All clues have been integrated:

[0168] ;

[0169] Corresponding hot branching clue features It was calculated in the same way.

[0170] TCM's spatial context modeling extracts multi-scale spatial information through convolutional kernels with different dilation rates. Compared to stacking large-scale convolutional layers, MSF requires only a small number of parameters to complete multi-scale feature representation, avoiding the high computational overhead of deep networks. TCM's channel dependency modeling uses pointwise convolution (…). Instead of a complex multi-head attention mechanism, TCM uses convolutional layers and sigmoid activation to achieve adaptive adjustment of channel weights. This structure has extremely low parameter count while maintaining strong feature selection capabilities, making it suitable for fast inference on edge devices. TCM's frequency cue modeling utilizes two-dimensional discrete cosine transform (DCT) to decompose features in the frequency domain, extracting complementary high- and low-frequency information without adding extra convolutional stacking. DCT is a deterministic operator that requires no training parameters, resulting in computational complexity far lower than deep network structures based on Fourier or wavelet transforms, naturally meeting lightweight design requirements. Through these optimizations, TCM achieves complementary fusion of spatial, channel, and frequency elements while significantly reducing the model's computational complexity and parameter count, enabling efficient integration with lightweight backbone networks and ensuring the overall method's deployability and real-time performance on edge devices.

[0171] In this embodiment, each decoder block consists of one Convolution followed by batch normalization and Activation function components: ; ; ;in, Indicates the first The significant prediction results of the layer decoder Indicates the first The significant prediction results of the layer decoder are upsampled after upsampling.

[0172] To enhance modal interaction and compensate for the performance degradation caused by model compression, this invention introduces two auxiliary losses: channel consistency loss and deep saliency supervision loss. In the lightweight RGB-T salient object detection method of this invention, the design of the loss function plays a crucial role in improving the performance of the lightweight model. Because lightweight networks are strictly constrained in terms of parameter count and computational complexity, their feature representation capabilities are often weaker than those of large-scale models, easily leading to problems such as insufficient modal alignment, missing boundary details, and insufficient utilization of shallow features. To alleviate these shortcomings, this invention introduces channel consistency loss and deep saliency supervision loss into the loss function, thereby effectively enhancing the discriminative ability and robustness of the lightweight model without increasing inference computational overhead.

[0173] The Channel Consistency Loss learns adaptive weights to align the channel features of RGB and thermal infrared modalities, constraining cross-modal features to maintain consistency across corresponding dimensions. This design compensates for insufficient cross-modal information fusion caused by the limited capacity of the lightweight backbone network, enabling the model to extract highly discriminative cross-modal complementary features with low computational overhead. Through this loss constraint, even in shallow convolutions or simplified structures, the texture information of the RGB modality and the contour information of the thermal infrared modality can be fully aligned, thereby improving detection accuracy and boundary integrity. On the other hand, the Deep Saliency Supervision Loss applies a supervisory signal to the output of the network's multi-layer decoder, allowing intermediate and shallow features to participate in the learning of salient targets. This multi-layer supervision significantly improves the sensitivity of shallow features to salient regions, avoiding the problem of insufficient shallow representation in lightweight networks. Simultaneously, the combined supervision of binary cross-entropy and intersection-over-union (IoU) further optimizes the detail representation of boundary regions while maintaining global mask accuracy. This design not only improves the overall robustness of the model but also ensures stable performance of small-scale networks in complex scenes.

[0174] The loss function is introduced only during the training phase and does not add any additional parameters or computational cost during the inference phase. Therefore, it enhances the expressive power of the lightweight model without affecting its running efficiency on edge devices. In other words, this invention indirectly enhances the cross-modal fusion capability and boundary detection accuracy of the lightweight network through reasonable loss function design, enabling the small model to approach or even reach the performance level of the large model while maintaining low computational consumption. This further strengthens the practical application value of this invention in edge deployment scenarios such as UAV inspection, intelligent security, and vehicle-mounted assisted driving.

[0175] Channel consistency loss calculates adaptive weights for each channel. To align RGB and thermal imaging features: ;in, Learning is achieved through two fully connected layers. This indicates the total number of channels corresponding to the current RGB and thermal imaging features.

[0176] Deep significance supervision loss incorporates binary cross-entropy and intersection , represented as:

[0177] ;in, Indicates the first The true label of the layer.

[0178] Total loss is defined as: .

[0179] In summary, this invention not only achieves deep fusion of cross-modal features through structural design in its overall technical solution, but also systematically optimizes lightweighting and edge adaptation. First, at the feature mining level, a cross-modal triple attention fusion mechanism (TCM) is introduced, employing low-overhead operators such as multi-scale dilated convolution, pointwise convolution, and discrete cosine transform (DCT) to model spatial context, channel dependence, and frequency semantics respectively. This effectively replaces traditional large-scale convolutions and complex multi-head attention structures, ensuring the capture of cross-modal details and complementary information while significantly reducing the number of parameters and computational complexity. Second, at the feature interaction level, a Local-Global Complementary Aggregation Module (LGCA) is designed. This module efficiently enhances local details through shallow convolution and bilinear upsampling, and uses global average pooling and lightweight fully connected layers to model global semantics. Compared to Transformer-based global attention, this significantly reduces computational overhead and memory usage while maintaining consistency and discriminability across different layers. Finally, at the training optimization level, the channel consistency loss and deep saliency supervision loss proposed in this invention can effectively compensate for the insufficient feature alignment and weakened shallow representation caused by the limited capacity of lightweight networks. This allows small-scale networks to still possess strong cross-modal fusion and boundary characterization capabilities without increasing inference overhead. The synergistic effect of these three factors results in an overall network containing only about 8.39M parameters, achieving near real-time inference speed while maintaining high-precision detection performance. It also enables efficient and stable operation on edge devices such as mobile terminals, embedded AI chips, and drones, thus providing a lightweight and deployable solution for the practical application of salient object detection.

[0180] Based on the above embodiments, this invention evaluates its SCF-Net on VT821, VT1000, and VT5000 by comparing it with eight RGB-T salient object detection algorithms, including CPD, SGDL, MMCI, FMCF, ADF, MIDD, ECFFNet, and CSRNet. The evaluation metrics include... , , and The data augmentations used in the experiments included flipping, rotating, and scale jittering. Training was performed using Adam (…). The learning rate and batch size are 4, and BatchNorm is synchronized on RTX3090.

[0181] Table 1 shows the quantitative performance of SCF-Net compared to other methods. SCF-Net achieved first place in 9 out of 12 metrics and second place in 1. This confirms its robustness and accuracy, highlighting the advantages of the spatiotemporal-frequency fusion method in a lightweight design.

[0182] Table 1 provides quantitative comparisons on the VT821, VT1000, and VT5000 datasets.

[0183]

[0184] As shown in Table 2, compared to CPD, ADF, MIDD, and CSRNet, SCF-Net achieves the lowest MAE with only 8.39M parameters and 9.13 GFLOPs, significantly outperforming previous methods in both accuracy and efficiency. Compared to CSRNet, it reduces computation by 13% while improving performance, demonstrating a better balance between accuracy and real-time capabilities.

[0185] Table 2 Comparison of Model Complexity and Performance

[0186] Methods CPD ADF MIDD CSRNet Ours MAE 0.046 0.048 0.043 0.044 0.035 Params(M) 29.233 83.129 52.428 1.011 8.389 FLOPs(G) 118.765 382.907 433.433 10.545 9.132

[0187] The lightweight RGB-T salient target detection method proposed in this invention, supporting edge devices, has significant advantages in cross-modal feature fusion, model lightweighting, and practical application adaptability. Firstly, in terms of cross-modal feature fusion, the Triple Clue Mining (TCM) module designed in this invention jointly models spatial structure, channel dependence, and frequency semantics, which can not only extract fine-grained information within a modality but also fully capture the deep complementary relationship between RGB and thermal infrared modalities. Combined with the Local-Global Complementary Aggregation (LGCA) module, it effectively enhances the consistency between local texture and global semantics, achieving efficient information interaction across levels and modalities. Compared with existing methods, this invention can generate more complete and clearer boundary salient target masks, exhibiting stronger robustness and generalization ability in complex backgrounds, low-light conditions, and camouflage environments. Secondly, in terms of lightweighting and edge adaptability, this invention uses MobileNetV2 as the backbone network and significantly reduces computational and storage requirements through depthwise separable convolutions and inverse residual structures. The entire network contains only about 8.39M parameters, resulting in computational overhead far lower than mainstream RGB-T detection methods based on ResNet or VGG backbones. Experiments show that this invention achieves near real-time inference speed while maintaining high accuracy, making it suitable for training and inference on GPU platforms, and also possessing the potential to run on embedded AI chips (such as NVIDIA Jetson and Huawei Ascend Atlas) and mobile terminals. Compared with existing high-energy-consuming models, this invention significantly reduces the computational and energy consumption pressure on edge devices, improving its practicality in edge scenarios such as drone inspection, intelligent security, vehicle-mounted assisted driving, and IoT monitoring. Finally, in terms of overall performance, this invention not only outperforms mainstream methods on publicly available RGB-T datasets such as VT821, VT1000, and VT5000, achieving the best results in 9 out of 12 metrics, but also maintains high inference efficiency. It achieves a balance between accuracy, speed, and resource consumption, proving that this method can guarantee detection quality and meet the needs of edge deployment in practical applications, thus possessing high technical value and promising prospects for widespread application.

[0188] The lightweight RGB-T salient target detection method proposed in this invention, supporting edge devices, has significant advantages in cross-modal feature fusion, model lightweighting, and practical application adaptability. Regarding cross-modal feature fusion, the Triple Clue Mining (TCM) module designed in this invention jointly models spatial structure, channel dependence, and frequency semantics. This not only extracts fine-grained information within a modality but also fully captures the deep complementary relationship between RGB and thermal infrared modes. Furthermore, by decomposing high- and low-frequency subbands through discrete cosine transform, it effectively enhances the boundary clarity and structural integrity of salient targets. Combined with the Local-Global Complementary Aggregation (LGCA) module, it effectively enhances the consistency between local texture and global semantics, achieving efficient information interaction across levels and modalities. Compared with existing methods, this invention can generate more complete and clearer salient target masks, exhibiting stronger robustness and generalization ability in complex backgrounds, low-light conditions, and camouflage environments. Secondly, regarding lightweight design and edge adaptability, this invention uses MobileNetV2 as the backbone network and significantly reduces computational and storage requirements through depthwise separable convolutions and inverse residual structures. The entire network contains only about 8.39M parameters, with computational overhead far lower than mainstream ResNet or VGG backbone RGB-T detection methods. Experiments show that this invention achieves near real-time inference speed while maintaining high accuracy, making it suitable for training and inference on GPU platforms, and also possessing the potential to run on embedded AI chips (such as NVIDIA Jetson and Huawei Ascend Atlas) and mobile terminals. Compared with existing high-energy-consuming models, this invention significantly reduces the computational and energy consumption pressure on edge devices, improving its practicality in edge scenarios such as drone inspection, intelligent security, vehicle-mounted assisted driving, and IoT monitoring. This invention achieves a balance between accuracy, speed, and resource consumption, proving that the method can guarantee detection quality and meet the needs of edge deployment in practical applications, thus possessing high technical value and promising prospects for promotion.

[0189] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0190] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0191] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0192] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0193] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A lightweight RGB-T salient target detection method, characterized in that, include: The RGB image to be detected and its corresponding thermal infrared image are acquired and input into the visible light feature extraction branch and thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and One thermal infrared feature; The first The visible light feature and the first The first thermal infrared feature is stitched together as the first... A multimodal fusion feature; Order No. The multimodal fusion features are sequentially processed through global average pooling and fully connected layers to obtain global context features; Global context features, the first The first multimodal fusion feature, and the second The first visible light feature or the first The first thermal infrared feature is input into the visible light feature optimization branch and the second thermal infrared feature optimization branch, respectively. In the local-global complementary aggregation module, long-range spatial features are extracted and compared with the first... The sum of the visible light joint features yields the first... The visible light fusion feature is used to extract long-range spatial features and the first visible light fusion feature. The first thermal infrared joint feature is obtained by adding the first feature together. One thermal infrared fusion feature; ; The first The first visible light fusion feature or the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature or the first The first thermal infrared branch cue feature includes: the first The first visible light fusion feature or the first The first thermal infrared fusion feature is fed into the spatial feature extraction branch of the triple clue mining module. After passing through the multi-scale fusion module, pooling, convolution, and activation, it is residually concatenated with the output of the multi-scale fusion module to output the second feature. The first visible light fusion feature or the first The spatial context features of the thermal infrared fusion feature; the first The first visible light fusion feature or the first The thermal infrared fusion features are fed into the channel feature extraction branch of the triple clue mining module, and after sequentially passing through pooling, two layers of convolution and activation, they are combined with the first... The first visible light fusion feature or the first The thermal infrared fusion feature is used for residual connection, and the output is the first... The first visible light fusion feature or the first The channel dependency features of the thermal infrared fusion feature; the first The first visible light fusion feature or the first The thermal infrared fusion features are fed into the frequency feature extraction branch of the triple clue mining module. The input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed. After aggregation by a multilayer perceptron, the summation is combined with the first... The first visible light fusion feature or the first The thermal infrared fusion feature is used for residual connection, and the output is the first... The first visible light fusion feature or the first The frequency cue features of the first thermal infrared fusion feature; the first The first visible light fusion feature or the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature or the first One thermal infrared branching cue feature; The first The first visible light branching cue feature, the first By splicing together the features of the thermal infrared branch cue, the first... A multimodal fusion feature; The first The visible light feature and the first After splicing the first thermal infrared feature, input the second The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit; For the The significant prediction result of the decoding unit is upsampled to obtain the first decoding unit. The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit; The salient prediction result of the first decoding unit is obtained as the salient target detection result of the RGB image to be detected; The local-global complementary aggregation module includes: a local complementary aggregation unit, which takes global context features as input, passes them sequentially through a self-attention mechanism, upsampling, and convolution, and outputs long-range spatial features; and a global complementary aggregation unit, which takes the first .... The first multimodal fusion feature, the first The first visible light feature or the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature or the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature or the first The first thermal infrared joint feature; the fusion unit combines the input long-range spatial features with the second thermal infrared joint feature; The first visible light joint feature or the first After adding the combined thermal infrared features, the output is the first... The first visible light fusion feature or the first A thermal infrared fusion feature.

2. The lightweight RGB-T salient target detection method according to claim 1, characterized in that, The RGB image to be detected and its corresponding thermal infrared image are acquired and input into the visible light feature extraction branch and thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and One thermal infrared feature, including: The RGB image to be detected will be obtained. and its corresponding thermal infrared image The first visible light feature is input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, and the first visible light feature output from the first convolutional sampling unit in each branch is obtained. With the first thermal infrared feature , represented as: ； ； The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , represented as: ； ； in, , This represents the total number of coding units in the visible light feature extraction branch or the thermal infrared feature extraction branch of the encoder in MobileNetV2. This represents depthwise separable convolution. Indicates downsampling; visible light characteristics Thermal infrared characteristics , , and These represent the height, width, and channels of the RGB image to be detected, respectively.

3. The lightweight RGB-T salient target detection method according to claim 2, characterized in that, Order No. The multimodal fusion features are sequentially processed through global average pooling and fully connected layers to obtain global context features; Taking global contextual features as input, the data sequentially passes through a self-attention mechanism, upsampling, and convolution to output long-range spatial features, including: For the Multimodal fusion features Perform global average pooling to obtain the global feature vector for each channel. ; A fully connected layer is used to perform a non-linear transformation on the global feature vector of each channel to obtain the corresponding channel attention weights. ; Using channel attention weights for the first The multimodal fusion features are weighted and adjusted to obtain the weighted features. ; The weighted features are concatenated with the channel features across all channels to obtain the global context features. ; Applying a self-attention mechanism to global context features to obtain long-range spatial features. ; in, Indicates the first The first multimodal fusion feature Each channel is located in eigenvalues ​​at that location , Indicates the first The total number of channels in a multimodal fusion feature; and Indicates learnable parameters, In the weighted feature graph, the first... Each channel is located in eigenvalues ​​at that location express convolution, Indicates upsampling, This represents the self-attention mechanism; This indicates a splicing operation.

4. The lightweight RGB-T salient target detection method according to claim 3, characterized in that, With the first The first multimodal fusion feature, the first The first visible light feature or the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature or the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature or the first The combined thermal infrared features include: The first The visible light branching cue features and the first By splicing together the features of the thermal infrared branch cue, the first... Multimodal fusion features , ; No. Visible light joint features , represented as: ； No. Joint thermal infrared features , represented as: ； in, Indicates the first Multimodal fusion features.

5. The lightweight RGB-T salient target detection method according to claim 4, characterized in that, Using the triple clue mining module to obtain the first The first visible light fusion feature and the first When considering the spatial context features, channel dependency features, and frequency cue features of thermal infrared fusion features, they all include: In the spatial feature extraction branch, the input features are sequentially processed by the multi-scale fusion module. Pooling ,convolution With activation Then, a residual connection is performed with the output of the multi-scale fusion module to output the corresponding spatial context features, represented as: ; In the channel feature extraction branch, the input features are sequentially processed through pooling, two convolutional layers, and activation, and then combined with the first... The first visible light fusion feature or the first The thermal infrared fusion features are used for residual connection, and the corresponding channel dependency features are output, represented as follows: ; In the frequency feature extraction branch, the input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and then summed. After aggregation by a multilayer perceptron, the summation is combined with the first sub-band. The first visible light fusion feature or the first The thermal infrared fusion features are residually concatenated to output the corresponding frequency cue features, represented as follows: ; in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light fusion feature; hour, Indicates the first One thermal infrared fusion feature; To express summation, This represents the two-dimensional discrete cosine transform.

6. The lightweight RGB-T salient target detection method according to claim 5, characterized in that, The first The first visible light fusion feature or the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature or the first The thermal infrared branching cue features are represented as follows: ； in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light branching cue feature; hour, Indicates the first One thermal infrared branching cue feature.

7. The lightweight RGB-T salient target detection method according to claim 6, characterized in that, For the The significant prediction result of the decoding unit is upsampled to obtain the first decoding unit. The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit include: The first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results ; The first The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... Features of fusion clues ; The first Features of fusion clues With the Upsampling results Enter the number together The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , represented as: ; in, , This indicates the total number of decoding units in the decoder of MobileNetV2.

8. The lightweight RGB-T salient target detection method according to claim 7, characterized in that, The total loss function is constructed and the encoder, local-global complementary aggregation module, triple clue mining module, and decoder are jointly trained; the total loss function is expressed as: ； Channel consistency loss ; Deep significance monitoring loss ; in, and They represent the first The visible light feature and the first One thermal infrared feature, Indicates the first Significant prediction results for each decoding unit, Indicates the first The actual tag corresponding to each decoding unit Indicates channel Channel attention weights, Indicates the first The total number of channels corresponding to each decoding unit and Indicates the first The visible light feature and the first The first thermal infrared feature Characteristics of each channel Denotes the square of the L2 norm. Represents the binary cross-entropy. This indicates intersection, union, and ratio.

Images