Lightweight RGB-T salient target detection method

Abstract

The invention relates to the technical field of computer vision, and discloses a lightweight RGB-T salient target detection method, which is characterized in that a local-global complementary aggregation module is designed on a feature interaction level, local detail enhancement is completed through shallow convolution and bilinear up-sampling, global semantic modeling is realized by using global average pooling and a full connection layer, and a significant target is obtained. The operation overhead and memory occupation are reduced, and the consistency and discrimination of cross-level information are kept at the same time; a triple clue mining module is introduced in a feature mining level, low-overhead operators such as multi-scale expansion convolution, point-by-point convolution and discrete cosine transform are adopted, space context, channel dependence and frequency semantics are modeled, capture of cross-modal details and complementary information is guaranteed, and the parameter quantity and calculation complexity are reduced; in the training optimization level, the problems of insufficient feature alignment and shallow expression weakening caused by limited lightweight network capacity are effectively compensated by using channel consistency loss and depth significance supervision loss, and the model detection capability is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly to a lightweight RGB-T salient object detection method. BACKGROUND

[0002] Salient object detection (SOD) is an important research direction in the field of computer vision, which aims to simulate the human visual attention mechanism, and automatically highlight and detect the most salient object region from the input image through automatic means. With the development of artificial intelligence and deep learning, salient object detection has wide application value in intelligent monitoring, automatic driving, unmanned aerial vehicle inspection, medical image analysis and other scenes. Early salient object detection methods mainly rely on low-level features and prior rules, such as color, texture, contrast and spatial distribution feature calculation, combined with center prior, background prior and other heuristic methods. This kind of method can achieve certain effect in simple environment, but in complex background, low illumination, occlusion and camouflage and other real application conditions, its robustness and generalization ability are seriously insufficient, which leads to poor detection accuracy of early salient object detection methods in real application conditions.

[0003] The rise of deep learning has promoted salient object detection methods based on convolutional neural networks, which can extract multi-scale semantic information end-to-end and significantly improve detection accuracy and stability using attention mechanism and feature fusion strategy. However, relying only on a single RGB modality still has performance degradation problems in insufficient light, cluttered background or cross-scene conditions. To solve this problem, multi-modal salient object detection has gradually attracted attention. RGB-D methods supplement spatial geometric information through depth maps, while RGB-T methods fuse visible light and thermal infrared modalities, and use their complementary characteristics to show higher robustness in night, low light and complex environments. At the same time, with the development of artificial intelligence from the cloud to the terminal and edge side, salient object detection also faces new development trends: how to realize lightweight and high efficiency on edge devices. Traditional deep models often have large number of parameters and complex calculations, which are difficult to realize real-time inference on mobile terminals, embedded AI chips, unmanned aerial vehicles and Internet of Things devices. Therefore, combining lightweight network design and cross-modal efficient fusion mechanism to develop an RGB-T salient object detection method with high precision and low power platform demand has become an important research direction and urgent need in this field.

[0004] In the RGB-based salient object detection method, J. Zhao et al. proposed a salient object detection method based on optical remote sensing image, which adopted recursive adaptive graph reasoning network (RAGRN) and region and boundary interaction mechanism. This method effectively enhanced the fusion of target region and boundary information, and improved the target detection ability in complex remote sensing images. Z. Wu et al. proposed a cascaded partial decoder structure (Cascaded Partial Decoder), which processed features in stages, not only accelerated the calculation process of salient object detection, but also improved the detection accuracy, especially suitable for real-time processing application scenarios. The existing RGB-based salient object detection method can achieve good performance under ideal lighting and clear background conditions, but in real environments such as low light, complex background and target occlusion, the detection results are often unstable and difficult to meet the needs of complex application scenarios.

[0005] To improve the stability of salient object detection results, researchers have proposed RGB-T salient object detection methods to improve detection accuracy by fusing the complementary information of visible light and thermal infrared modalities. In the RGB-T salient object detection method, Q. Zhang et al. proposed a method for RGB-T salient object detection by fusing multi-layer convolutional neural network (CNN) features; this method effectively improves the salient object detection accuracy in RGB-T images by fusing multi-scale features of RGB and infrared (T) images. Z. Tu et al. proposed an RGB-T salient object detection method based on collaborative graph learning; by constructing a collaborative graph, the feature relationship between RGB and infrared modalities is dynamically adjusted, significantly enhancing the target detection ability in complex backgrounds. H. Zhou et al. proposed an RGB-T salient object detection method based on location-aware relationship learning; this method improves the accuracy and stability of target detection by learning the relationship between target location and multi-modal features, especially in dynamic environments. F. Huo et al. proposed an efficient context-guided stacked refinement network (CSRN), which effectively improves the accuracy and efficiency of RGB-T salient object detection by guiding network to fuse context information, especially in low light and complex environments. W. Zhou et al. proposed an effective and consistent feature fusion network (ECFFNet), which improves the accuracy of RGB-T salient object detection by optimizing the feature fusion strategy, especially in the application of cross-modal feature fusion and consistency loss function. Z. Tu et al. proposed a multi-interaction dual decoder (MIDD) model, which strengthens the collaborative effect between RGB and infrared image modalities through a dual decoder structure and multi-interaction mechanism, significantly improving detection accuracy.

[0006] Although the existing RGB-T salient object detection method has made progress in accuracy, most of the existing methods only focus on the feature interaction in the spatial domain or the channel domain, ignoring the complementary role of the frequency domain semantics, leading to insufficient cross-modal feature alignment and fusion, making it difficult to fully play the complementary advantages of RGB and T modalities, and further unable to obtain more rich image features, and unable to achieve higher precision salient object detection. And most of the mainstream RGB-T detection models rely on large-scale backbone networks such as ResNet and VGG, and superimpose complex multi-layer feature fusion structures. Although this design improves the accuracy, it brings extremely high computational cost and storage overhead, and the efficiency of salient object detection is low, which is difficult to promote in real-time applications. In addition, with the AI application gradually moving from the cloud to the terminal and the edge, actual application scenarios (such as unmanned aerial vehicle inspection, intelligent security, vehicle-mounted equipment) have higher requirements for lightweight and real-time; However, the existing RGB-T salient object detection method has not been optimized in terms of parameter size, computational delay and energy consumption for edge devices, making it difficult to realize fast reasoning and efficient operation on embedded AI chips, mobile terminals or Internet of Things devices, and further unable to realize deployment in edge devices, limiting the application scenarios.

[0007] In summary, the existing technology has deficiencies in cross-modal feature fusion, which cannot effectively fuse the features extracted from RGB images and thermal infrared images, and is prone to lose key features during fusion, resulting in insufficient fusion, low target detection accuracy, and unstable detection results. To improve detection accuracy, the existing technology relies on large-scale backbone networks for feature fusion, which brings extremely high computational cost and high storage overhead, cannot realize lightweight deployment in edge devices, and has low detection efficiency for salient objects. SUMMARY

[0008] Therefore, the technical problem to be solved by the present application is to overcome the problem of low target detection accuracy due to insufficient multi-modal feature fusion caused by ignoring the frequency domain semantics in the prior art, and the problem of high computational cost and low detection efficiency caused by relying on large-scale backbone networks.

[0009] To solve the above technical problems, the present application provides a lightweight RGB-T salient object detection method, comprising: obtaining an RGB image to be detected and its corresponding thermal infrared image, inputting them into the visible light feature extraction branch and the thermal infrared feature extraction branch of the encoder of MobileNetV2 respectively, obtaining visible light features and thermal infrared features; splicing the first visible light feature and the first thermal infrared feature as the first multi-modal fusion feature; let the first The global context feature is sequentially input into a global average pooling layer and a fully connected layer to obtain a global context feature; The global context feature, the first multi-modal fusion feature, and the first visible light feature and the first thermal infrared feature are input into the first local-global complementary aggregation module in the visible light feature optimization branch / thermal infrared feature optimization branch to extract long-distance spatial features and the first visible light joint feature and the first thermal infrared joint feature, and are added to obtain the first visible light fusion feature and the first thermal infrared fusion feature; ; The first visible light fusion feature and the first thermal infrared fusion feature are input into the triple clue mining module and are sequentially input into a spatial feature extraction branch, a channel feature extraction branch, and a frequency feature extraction branch to obtain spatial context features, channel dependency relationship features, and frequency clue features, which are added to obtain the first visible light branch clue feature and the first thermal infrared branch clue feature; The first visible light branch clue feature and the first thermal infrared branch clue feature are spliced to obtain the first multi-modal fusion feature; The first visible light feature and the first thermal infrared feature are spliced and input into the first decoding unit to obtain the first decoding unit; The saliency prediction result of the first decoding unit is up-sampled to obtain the first up-sampling result; the first visible light branch clue feature and the first thermal infrared branch clue feature are added to obtain the first fusion clue feature, which is input into the first decoding unit together with the first up-sampling result to obtain the saliency prediction result of the first decoding unit; The saliency prediction result of the first decoding unit is obtained as the saliency target detection result of the to-be-detected RGB image.

[0010] Preferably, the RGB image to be detected and its corresponding thermal infrared image are acquired and input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and One thermal infrared feature, including: The RGB image to be detected will be acquired. and its corresponding thermal infrared image The first visible light feature is input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, and the first visible light feature is obtained from the output of the first convolutional sampling unit in the visible light feature extraction branch and the thermal infrared feature extraction branch. With the first thermal infrared feature , is represented as: ; ; The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , is represented as: ; ; in, , This represents the total number of coding units in the visible light feature extraction branch / thermal infrared feature extraction branch of the encoder in MobileNetV2. This represents depthwise separable convolution. Indicates downsampling; visible light characteristics Thermal infrared characteristics , , and These represent the height, width, and channels of the RGB image to be detected, respectively.

[0011] Preferably, the local-global complementary aggregation module includes: The local complementary aggregation unit takes global context features as input, passes through a self-attention mechanism, upsampling and convolution in sequence, and outputs long-range spatial features. Global complementary aggregation unit, with the first The first multimodal fusion feature, the first The first visible light feature / the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature / the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature / the first A combined thermal infrared feature; The fusion unit combines the input long-range spatial features, the first... The first visible light joint feature / the first After adding the combined thermal infrared features, the output is the first... The first visible light fusion feature / the first A thermal infrared fusion feature.

[0012] Preferably, let the first The multimodal fusion features are sequentially processed through global average pooling and fully connected layers to obtain global context features. Using these global context features as input, they are sequentially processed through a self-attention mechanism, upsampling, and convolution to output long-range spatial features, including: For the Multimodal fusion features Perform global average pooling to obtain the global feature vector for each channel. ; A fully connected layer is used to perform a non-linear transformation on the global feature vector of each channel to obtain the corresponding channel attention weights. ; Using channel attention weights for the first The multimodal fusion features are weighted and adjusted to obtain a weighted feature map. ; The weighted feature map is concatenated with the channel features of all channels to obtain the global context features. ; Applying a self-attention mechanism to global context features to obtain long-range spatial features. ; in, Indicates the first The first multimodal fusion feature Each channel is located in eigenvalues ​​at that location , Indicates the first The total number of channels in a multimodal fusion feature; and Indicates learnable parameters, In the weighted feature graph, the first... Each channel is located in eigenvalues ​​at that location express convolution, Indicates upsampling, This represents the self-attention mechanism.

[0013] Preferably, with the first The first multimodal fusion feature, the first The first visible light feature / the first Let the first thermal infrared feature be the input, and let the second... The first visible light feature / the first The first thermal infrared feature, after undergoing a self-attention mechanism and convolution, is compared with the second feature after upsampling and convolution. The multimodal fusion features are added together to output the first feature. The first visible light joint feature / the first The combined thermal infrared features include: The first The visible light branching cue features and the first By splicing together the features of the thermal infrared branch cue, the first... Multimodal fusion features , ; No. Visible light joint features , is represented as: ; No. Joint thermal infrared features , is represented as: ; in, Indicates the first A multimodal fusion feature; This indicates a splicing operation.

[0014] Preferably, the first The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature includes: The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the spatial feature extraction branch of the triple clue mining module. After passing through the multi-scale fusion module, pooling, convolution and activation in sequence, they are residually connected with the output of the multi-scale fusion module to output spatial context features. The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the channel feature extraction branch of the triple clue mining module, and after sequentially passing through pooling, two layers of convolution and activation, they are combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output channel dependency features; The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the frequency feature extraction branch of the triple clue mining module. The input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed. After aggregation by a multilayer perceptron, the summation is combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output frequency cue features; The first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The visible light branching cue features and the first One thermal infrared branching cue feature.

[0015] Preferably, the triple clue mining module is used to obtain the first... The first visible light fusion feature and the first When considering the spatial context features, channel dependency features, and frequency cue features of thermal infrared fusion features, they all include: In the spatial feature extraction branch, the input features are sequentially processed by the multi-scale fusion module. Pooling ,convolution With activation Then, a residual connection is performed with the output of the multi-scale fusion module to output the corresponding spatial context features, represented as: ; In the channel feature extraction branch, the input features are sequentially processed through pooling, two convolutional layers, and activation, and then combined with the first... The first visible light fusion feature / the first The thermal infrared fusion features are used for residual connection, and the corresponding channel dependency features are output, represented as follows: ; In the frequency feature extraction branch, the input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and then summed. After aggregation by a multilayer perceptron, the summation is combined with the first sub-band. The first visible light fusion feature / the first The thermal infrared fusion features are residually concatenated to output the corresponding frequency cue features, represented as follows: ; in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light fusion feature; hour, Indicates the first One thermal infrared fusion feature; To express summation, This represents the two-dimensional discrete cosine transform.

[0016] Preferably, the first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature / the The thermal infrared branching cue features are represented as follows: ; in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light branching cue feature; hour, Indicates the first One thermal infrared branching cue feature.

[0017] Preferably, for the first Upsample the significant prediction result of the decoding unit to obtain the first... The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the nth decoding unit. Significant prediction results for each decoding unit include: The first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results ; The first The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... Features of fusion clues ; The first Features of fusion clues With the Upsampling results Enter the number together The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , is represented as: ; in, , This indicates the total number of decoding units in the decoder of MobileNetV2.

[0018] Preferably, a total loss function is constructed for joint training of the encoder, the local-global complementary aggregation module, the triple clue mining module, and the decoder; the total loss function is expressed as: ; Channel consistency loss ; Deep significance monitoring loss ; in, and They represent the first The visible light feature and the first One thermal infrared feature, Indicates the first Significant prediction results for each decoding unit, Indicates the first The actual tag corresponding to each decoding unit Indicates channel Channel attention weights, Indicates the first The total number of channels corresponding to each decoding unit and Indicates the first The visible light feature and the first The first thermal infrared feature Characteristics of each channel Denotes the square of the L2 norm. Represents the binary cross-entropy. This indicates intersection, union, and ratio.

[0019] Compared with the prior art, the above-described technical solution of the present invention has the following advantages:

[0020] The lightweight RGB-T salient object detection method described in this invention uses MobileNetV2 as the backbone of the RGB-T stream. MobileNetV2 reduces computational cost through depthwise separable convolutions and enhances feature representation and gradient flow by combining inverse residual structures and linear bottlenecks, achieving both lightweight and high efficiency while maintaining accuracy, enabling real-time detection in edge devices. Simultaneously, this invention introduces a local-global complementary aggregation module between the encoder and decoder. By upsampling and convolutionally performing multimodal fusion features, it adds them to features that have undergone self-attention and convolution to obtain joint features, achieving the fusion of local image texture and cross-modal details. Compared to multi-layer convolution stacking, this significantly reduces parameters and computational cost, improving salient object detection efficiency. Global average pooling and small-scale fully connected layers are used to extract contextual semantics, constructing global contextual features from which long-range spatial features are extracted, avoiding the high computational overhead of large-scale attention networks and improving salient object detection efficiency while maintaining detection accuracy. The joint features are then added to the long-range spatial features to obtain the corresponding fusion features for subsequent decoding. The local-global complementary aggregation module effectively integrates local texture, global semantics, and modality-aware features to enhance cross-layer information consistency and semantic alignment. Simultaneously, the module maintains modularity and pluggability, allowing for flexible embedding into lightweight backbone networks. This significantly reduces model complexity while enhancing the interaction between local and global features, ensuring cross-layer information consistency and discriminative feature extraction. Furthermore, it enables the invention to maintain efficient inference and real-time response capabilities on edge devices, further improving detection accuracy and real-time performance in application scenarios with edge devices, and enhancing the efficiency of salient target detection.

[0021] The lightweight RGB-T salient object detection method provided by the application further introduces a triple clue mining module between the encoder and the decoder to realize spatial, channel and frequency triple complementary fusion. When spatial features are extracted, multi-scale spatial information is extracted through convolution kernels with different inflation rates. Compared with stacked large-scale convolution layers, multi-scale feature expression can be completed with only a small amount of parameters, avoiding high computational overhead caused by deep networks, increasing feature extraction scale, improving feature extraction accuracy, and speeding up salient object detection; when channel features are extracted, point-by-point convolution and Sigmoid activation are used instead of complex multi-head attention mechanism to realize adaptive adjustment of weights between channels, with extremely low parameter quantity and strong feature screening ability, suitable for fast reasoning of edge devices, and can be deployed on edge devices; when frequency features are extracted, two-dimensional discrete cosine transform is used to decompose the features in the frequency domain, which can extract high and low frequency complementary information without increasing additional convolution stacking, and the two-dimensional discrete cosine transform is a deterministic operator that does not require training parameters, with a much lower computational complexity than deep network structures based on Fourier or wavelet, effectively avoiding parameter inflation and computational redundancy caused by complex attention networks, realizing model lightweight, and further improving salient object detection efficiency. The triple clue mining module jointly explores the salient clues in the space, channel and frequency domain, so as to extract intra-modal details and cross-modal complementary information, improve the feature extraction progress of the image, and ensure the accuracy of salient object detection; at the same time, through modular design, it can be flexibly embedded in a lightweight backbone network, keeping the high-precision cross-modal feature fusion capability while controlling the parameter quantity and computational complexity within the range that can be tolerated by edge devices, improving the salient object detection efficiency, and ensuring the deployability and real-time performance of the overall method on edge devices.

[0022] This application employs channel consistency loss and deep saliency supervision loss during model training, effectively enhancing the discriminative power and robustness of the lightweight model without increasing inference computational overhead. Channel consistency loss learns adaptive weights to align the channel features of RGB and thermal infrared modalities, constraining cross-modal features to maintain consistency across corresponding dimensions. This compensates for insufficient cross-modal information fusion caused by the limited capacity of the lightweight backbone network, enabling the model to extract highly discriminative cross-modal complementary features with low computational cost. Through this loss constraint, even with shallow convolutions or simplified structures, the texture information of the RGB modality and the contour information of the thermal infrared modality can be fully aligned, thereby improving detection accuracy and boundary integrity. Deep saliency supervision loss applies a supervisory signal to the output of the network's multi-layer decoder, enabling intermediate and shallow features to participate in the learning of salient targets. This multi-layer supervision significantly improves the sensitivity of shallow features to salient regions, avoiding the problem of insufficient shallow representation in lightweight networks. Simultaneously, the combined supervision of binary cross-entropy and cross-union ratio (CUN) further optimizes the detail representation of boundary regions while maintaining global mask accuracy. This not only improves the overall robustness of the model but also ensures the stability of small-scale networks in complex scenarios, further enhancing the accuracy of salient target detection in the trained model and improving the stability of salient target detection in different application scenarios. Attached Figure Description

[0023] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart of the lightweight RGB-T salient target detection method provided by the present invention; Figure 2 This is a schematic diagram of a local-global complementary aggregation module; Figure 3 This is a schematic diagram of the triple clue mining module. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0025] Reference Figure 1 As shown in the flowchart, the lightweight RGB-T salient target detection method provided by the present invention includes the following specific steps: Acquire the RGB image to be detected and its corresponding thermal infrared image The features are input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, to obtain... Visible light characteristics and thermal infrared features ; The first The visible light feature and the first The first thermal infrared feature is stitched together as the first... Multimodal fusion features Order No. Multimodal fusion features Global context features are obtained by sequentially passing through global average pooling and fully connected layers. ; global context features , No. Multimodal fusion features With the Visible light characteristics / th thermal infrared features The first branch in the input visible light feature optimization branch / thermal infrared feature optimization branch In the local-global complementary aggregation module, long-range spatial features are extracted and compared with the first... The first visible light joint feature / the first The first thermal infrared joint feature is added together to obtain the second feature. Visible light fusion features / th One thermal infrared fusion feature ; ; The first Visible light fusion features / th One thermal infrared fusion feature The input is processed by a triple clue mining module, which extracts spatial context features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. Channel dependency characteristics Frequency cue features Add them together to get the first one. Visible light branching clue features / th Features of thermal infrared branch clues ; The first Visible light branching clue features , No. Features of thermal infrared branch clues splice together to obtain the first A multimodal fusion feature; The first Visible light characteristics With the thermal infrared features After splicing, enter the first The decoding unit sequentially undergoes convolution, batch normalization, and activation functions to output the nth decoding unit. Significant prediction results for each decoding unit ; For the first Significant prediction results for each decoding unit Perform upsampling to obtain the first Upsampling results ; will the first Visible light branching clue features With the Features of thermal infrared branch clues Add them together to get the first one. Features of fusion clues , and the Upsampling results Enter the first The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit ; Obtain the significant prediction result of the first decoding unit. This serves as the result of salient target detection in the RGB image to be detected.

[0026] The local-global complementary aggregation module includes: Local complementary aggregation units, with global context features The input is processed sequentially through a self-attention mechanism, upsampling, and convolution, outputting long-range spatial features. ; Global complementary aggregation unit, with the first Multimodal fusion features , No. Visible light characteristics / th thermal infrared features As input, let the first... Visible light characteristics / th thermal infrared features After self-attention and convolution, compared with the first convolution after upsampling... Multimodal fusion features Add them together and output the first result. Visible light joint features / th Joint thermal infrared features ; The fusion unit integrates the long-range spatial features of the input. , No. Visible light joint features / th Joint thermal infrared features After adding, output the first... Visible light fusion features / th One thermal infrared fusion feature .

[0027] Specifically, the visible light feature extraction branch and the thermal infrared feature extraction branch of the encoder based on MobileNetV2 obtain... Visible light characteristics and One thermal infrared feature, including: The RGB image to be detected will be acquired. and its corresponding thermal infrared image The first visible light feature is input into the visible light feature extraction branch and the thermal infrared feature extraction branch of the MobileNetV2 encoder, respectively, and the first visible light feature output from the first convolutional sampling unit in each branch is obtained. With the first thermal infrared feature , is represented as: ; ; The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , is represented as: ; ; in, , This represents the total number of coding units in the visible light feature extraction branch / thermal infrared feature extraction branch of the encoder in MobileNetV2. This represents depthwise separable convolution. Indicates downsampling; visible light characteristics Thermal infrared characteristics , , and respectively represent the height, width and channel of the RGB image to be detected.

[0028] In this embodiment, feature concatenation refers to transverse connection of two features along the channel dimension; feature addition refers to element-wise addition of pixels on two feature maps corresponding to two features.

[0029] Referring to Figure 2 , it is a schematic diagram of a local-global complementary aggregation module, and the local-global complementary aggregation module in this embodiment includes:

[0030] ①Obtain long-distance spatial features : Perform global average pooling on the first multi-modal fusion feature to obtain a global feature vector of each channel ; Use a fully connected layer to perform nonlinear transformation on the global feature vector of each channel to obtain a corresponding channel attention weight ; Use the channel attention weight to adjust the first multi-modal fusion feature ; Concatenate the weighted features in all channels to obtain a global context feature ; Apply a self-attention mechanism to the global context feature to obtain long-distance spatial features ; wherein represents the feature value of the first channel in the first multi-modal fusion feature at position , , represents the total number of channels in the first multi-modal fusion feature; and represent learnable parameters, represents the feature value of the first channel in the weighted feature at position , represents convolution, represents up-sampling, represents a self-attention mechanism;

[0031] ②Obtain visible light joint feature / thermal infrared joint feature : Combine the first visible light branch clue feature with the first By splicing together the features of the first thermal infrared branch cue, the first... Multimodal fusion features , ; No. Visible light joint features , is represented as: ; No. Joint thermal infrared features , is represented as: ; in, Indicates the first A multimodal fusion feature; This indicates a splicing operation.

[0032] Reference Figure 3 The diagram shown is a schematic of the triple clue mining module; in this embodiment of the invention, the first... The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature includes: The first Visible light fusion features / th One thermal infrared fusion feature The spatial feature extraction branch, fed into the triple clue mining module, passes sequentially through a multi-scale fusion module, pooling, convolution, and activation. It is then residually concatenated with the output of the multi-scale fusion module to output spatial context features. ; The first Visible light fusion features / th One thermal infrared fusion feature The channel feature extraction branch fed into the triple clue mining module undergoes pooling, two convolutional layers, and activation sequentially, before being connected to the first... The first visible light fusion feature / the first Residual connections are performed on the thermal infrared fusion features to output channel dependency features. ; The first Visible light fusion features / th One thermal infrared fusion feature The input features are fed into the frequency feature extraction branch of the triple clue mining module. After being decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed, they are aggregated by a multilayer perceptron and then compared with the first... The first visible light fusion feature / the first Residual connections are performed on the thermal infrared fusion features to output frequency cue features. ; The first Visible light fusion features / th One thermal infrared fusion feature Spatial context features Channel dependency characteristics Frequency cue features Add them together to get the first one. Visible light branching clue features With the Features of thermal infrared branch clues , is represented as: ; in, , Time represents visible light. Time indicates thermal infrared; hour, Indicates the first One visible light branching cue feature; hour, Indicates the first One thermal infrared branching cue feature.

[0033] This embodiment utilizes a triple clue mining module to obtain the first... Visible light fusion features and the One thermal infrared fusion feature Spatial context features Channel dependency characteristics Frequency cue features At times, all include: In the spatial feature extraction branch, the input features are sequentially processed by the multi-scale fusion module. Pooling ,convolution With activation Then, the corresponding spatial context features are output. , is represented as: ; In the channel feature extraction branch, the input features are sequentially processed through pooling, two convolutional layers, and activation, and the corresponding channel dependency features are output. , is represented as: ; In the frequency feature extraction branch, the input features are decomposed into four directional subbands by two-dimensional discrete cosine transform and summed, and then aggregated by a multi-layer perception to output the corresponding frequency cue features , which is represented as: ; wherein, , represents visible light, represents thermal infrared; , represents the th visible light fusion feature; , represents the th thermal infrared fusion feature; represents summation, represents two-dimensional discrete cosine transform.

[0034] After introducing the triple feature mining module in the present example, the th visible light branch cue feature and the th thermal infrared branch cue feature are spliced to obtain the th multi-modal fusion feature . With the th multi-modal fusion feature , the th visible light feature / the th thermal infrared feature as input, the th visible light feature / the th thermal infrared feature is subjected to self-attention mechanism and convolution, and then added to the th multi-modal fusion feature which is subjected to up-sampling and convolution to output the th visible light joint feature / the th thermal infrared joint feature . The th visible light branch cue feature is added to the th thermal infrared branch cue feature to obtain the th fusion cue feature , which is input with the up-sampling result of the th decoding unit significant prediction result to obtain the The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , is represented as: .

[0035] Among them, when At that time, the first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results .

[0036] Based on the above embodiments, in this embodiment of the invention, a total loss function is constructed to jointly train the encoder, the local-global complementary aggregation module, the triple clue mining module, and the decoder; the total loss function is expressed as: ; Channel consistency loss ; Deep significance monitoring loss ; in, and They represent the first The visible light feature and the first One thermal infrared feature, Indicates the first Significant prediction results for each decoding unit, Indicates the first The actual tag corresponding to each decoding unit Indicates channel Channel attention weights, Indicates the first The total number of channels corresponding to each decoding unit and Indicates the first The visible light feature and the first The first thermal infrared feature Characteristics of each channel Denotes the square of the L2 norm. Represents the binary cross-entropy. This indicates intersection, union, and ratio.

[0037] In summary, the present application proposes a lightweight RGB-T salient object detection method (SCF-Net) supporting edge devices. SCF-Net jointly explores spatial structure, channel dependency and frequency semantics to realize deep cross-modal feature fusion. Specifically, the present application designs a triple clue mining (TCM) module based on discrete cosine transform (DCT), which effectively integrates spatial structure, channel dependency and frequency semantics, thereby extracting intra-modal details and cross-modal complementary information; in addition, the present application introduces a local-global complementary aggregation (LGCA) module to integrate local texture and global semantics, enhance cross-layer semantic alignment and information consistency; with the help of a lightweight backbone network, SCF-Net contains only 8.39M parameters, supporting real-time inference.

[0038] Based on the above embodiment, in the present embodiment, a local-global complementary aggregation module and a triple clue mining module are introduced simultaneously to detect salient objects; the local-global complementary aggregation module effectively fuses local texture, global semantics and modal perception features to enhance cross-layer information consistency and semantic alignment; the triple clue mining module is used to jointly explore spatial, channel and frequency domain salient clues, thereby extracting intra-modal details and cross-modal complementary information.

[0039] The present application adopts MobileNetV2 as the backbone of the RGB-T stream. MobileNetV2 reduces the computational load through deep separable convolution, combines the inverted residual structure and linear bottleneck to improve feature expression and gradient flow, and realizes lightweight and high efficiency while ensuring accuracy.

[0040] Referring to Figure 1 , five encoding units are arranged in the encoder, and in each encoding unit, visible light features and thermal infrared features are extracted through deep separable convolution and downsampling operation , denoted as: ; ; wherein, represents the height of the input RGB picture, represents the width of the input RGB picture, represents the channel of the input RGB picture, represents the input RGB picture, represents the input thermal infrared picture.

[0041] The Local-Global Complementary Aggregation (LGCA) module comprises Local Complementary Aggregation (LCA) and Global Complementary Aggregation (GCA), which together enhance fine-grained details and overall semantics. In terms of lightweight design, the LGCA module also fully considers the computational constraints of edge devices. LCA achieves the fusion of local textures and cross-modal details through shallow convolutions and upsampling operations, and aligns and enhances details of RGB and infrared features through upsampling and a small number of convolutions. Compared to traditional multi-layer deep convolutional structures, LCA only requires a single bilinear interpolation and a 3×3 convolution to complete local texture fusion, thus effectively reducing the number of convolutional kernels and computational overhead. It utilizes self-attention to refine fine-grained features within a small scope, avoiding the high complexity of global attention and significantly reducing memory usage and computational costs. GCA employs global average pooling to extract contextual information, and then combines it with a lightweight fully connected layer to complete channel weight learning. Compared with global modeling methods based on Transformer or multi-head attention, its parameter count and computational complexity are significantly reduced, avoiding the high computational overhead of large-scale attention networks. At the same time, global semantic information is applied to the feature map in a one-time weighted manner, eliminating the need to repeatedly calculate the attention matrix and ensuring inference speed on edge devices. Meanwhile, the overall structure of LGCA maintains modularity and pluggability, allowing for flexible embedding of lightweight backbone networks, significantly reducing model complexity while enhancing the interaction between local and global features. This design ensures cross-layer information consistency and discriminative feature extraction, enabling the invention to maintain efficient inference and real-time response capabilities on edge devices.

[0042] In the RGB branch, LCA first fuses the features. Upsampling with Matching resolution, then through self-attention Refine To enhance object perception and differentiation, joint features are obtained. : ; ; in, This indicates a splicing operation. Indicates visible light branching cue characteristics, Indicating hot branching clue characteristics, the first The visible light branching cue features and thermal branching cue features of the layer are derived from the first layer. The TCM output of layer 5 yields the visible light branching cue features and thermal branching cue features of layer 5. and , express Convolution operation, This indicates 2× bilinear upsampling.

[0043] For GCA, global context features It effectively guides the weight adjustment of each channel in the feature map, and the global context From high-level features and China passed extract: ; in, First of all and The two feature maps are fused to obtain a new fused feature map. Next, the fused feature map Perform global average pooling to obtain the global feature vector for each channel. Then, a fully connected layer is used to process the global feature vector of each channel. Perform a nonlinear transformation to obtain the channel attention weights. Finally, the obtained channel attention weights are used. For fused feature maps After weighted adjustment, the weighted feature map is obtained. The generated weighted feature map As Operation output : ; ; ; ; ; ; in, Represents the fused feature map The Middle Each channel is located in The value at that location, , For learnable parameters, For learnable parameters, Represents the weighted feature map The Middle The spatial location of each channel The eigenvalue at that location. For Applying self-attention to model long-range spatial dependencies yields long-range spatial features. .

[0044] LGCA output visible light fusion characteristics , is represented as: .

[0045] Corresponding thermal fusion characteristics The same way is calculated.

[0046] In terms of lightweight design, the LGCA module is composed of local complementary aggregation (LCA) and global complementary aggregation (GCA), both of which ensure feature interaction capability while considering computational efficiency and parameter scale.

[0047] The triple feature mining module TCM of the application enhances visible light fusion features and thermal fusion features by jointly using spatial, channel and frequency clues, and adaptively emphasizes structure, semantics and texture to enhance saliency representation. In the lightweight design of the application, TCM not only realizes the joint modeling of spatial, channel and frequency triple clues, but also optimizes the structure for the computing power limit of edge devices. Specifically, TCM uses multi-scale convolution and low-impact operators such as discrete cosine transform (DCT) in design, effectively avoiding parameter expansion and computational redundancy caused by complex attention networks; at the same time, through modular design, TCM can be flexibly embedded in lightweight backbone networks (such as MobileNetV2), and does not rely on large-scale ResNet or Transformer structure. This mechanism not only maintains high-precision cross-modal feature fusion capability, but also controls the parameter quantity and computational complexity within the range that edge devices can bear, thereby ensuring the real-time inference performance and stability of the model on resource-constrained platforms.

[0048] Spatial context of TCM is extracted through a multi-scale fusion (MSF) module, and then and a convolution are performed: ; wherein the MSF first applies convolution kernels with different dilation rates to the input features to obtain multi-scale features ; then the multi-scale features are spliced and integrated using convolution to obtain the final output multi-scale fusion features : ; ; ; ; wherein represents convolution with dilation rate .

[0049] Channel dependency of TCM is captured through and two convolutions, represented as: ​ ; TCM frequency clues , The data is decomposed into four directional subbands {LL, LH, HL, HH} using a two-dimensional discrete cosine transform (DCT), and then... Aggregation, represented as: ; in, This represents the summation operation. For each channel, first calculate its two-dimensional DCT-II coefficients. Then let the vertical and horizontal cutoff frequency indices be respectively , Define four masks Four directional offspring are obtained by element-wise multiplying the channel coefficients with the four masks. : ; ; ; ; ; ; ; in, , , Definition and Similarly, for height, for width, for The number of channels, This indicates an element-wise dot product operation.

[0050] Ultimate visible light branching cue features All clues have been integrated: ; Corresponding hot branching clue features It was calculated in the same way.

[0051] TCM's spatial context modeling extracts multi-scale spatial information through convolutional kernels with different dilation rates. Compared to stacking large-scale convolutional layers, MSF requires only a small number of parameters to complete multi-scale feature representation, avoiding the high computational overhead of deep networks. TCM's channel dependency modeling uses pointwise convolution (…). The convolution is replaced by a sigmoid activation to realize adaptive adjustment of inter-channel weights, and the structure has extremely low parameter quantity while maintaining strong feature filtering capability, and is suitable for fast reasoning of edge devices. The frequency clue modeling of TCM uses two-dimensional discrete cosine transform (DCT) to perform frequency domain decomposition on the features, which can extract high and low frequency complementary information without increasing additional convolution stacking. DCT is a deterministic operator that does not require training parameters, and has much lower computational complexity than deep network structures based on Fourier or wavelet, and naturally meets the lightweight design requirements. Through the above optimization, TCM realizes the spatial, channel and frequency triple complementary fusion while significantly reducing the computational complexity and parameter quantity of the model, so that it can be efficiently combined with lightweight backbone networks, ensuring the deployability and real-time performance of the overall method on edge devices.

[0052] In the present embodiment, each decoder block is composed of one convolution followed by batch normalization and an activation function: ; ; ; wherein, represents the saliency prediction result of the layer decoder, represents the up-sampling result of the saliency prediction result of the layer decoder after up-sampling.

[0053] To enhance the interaction between modalities and compensate for the performance degradation caused by model compression, the present application introduces two auxiliary losses: channel consistency loss and deep saliency supervision loss. In the lightweight RGB-T salient object detection method of the present application, the design of the loss function plays a key role in improving the performance of the lightweight model. Due to the strict constraints on parameter quantity and computational complexity of lightweight networks, their feature expression capability is often weaker than that of large-scale models, and problems such as insufficient modal alignment, missing boundary details and insufficient use of shallow features are prone to occur. To alleviate these deficiencies, the present application introduces channel consistency loss and deep saliency supervision loss in the loss function, thereby effectively enhancing the discriminability and robustness of the lightweight model without increasing the inference computational overhead.

[0054] The channel consistency loss constrains the consistency of cross-modal features in the corresponding dimensions by learning adaptive weights to align the channel features of the RGB and thermal infrared modalities. This design can compensate for the insufficient cross-modal information fusion caused by the limited capacity of the lightweight backbone network, enabling the model to still extract discriminative cross-modal complementary features under low computational overhead. Through the constraint of this loss, even in shallow convolution or simplified structure, the texture information of the RGB modality and the contour information of the thermal infrared modality can be fully aligned, thereby improving the detection accuracy and boundary integrity. On the other hand, the deep saliency supervision loss imposes a supervision signal on the output of the multi-layer decoder of the network, enabling the intermediate and shallow features to participate in the learning of the salient target. This multi-level supervision method can significantly improve the sensitivity of shallow features to salient regions, avoiding the problem of insufficient expression of shallow layers in lightweight networks. At the same time, the combination of binary cross-entropy and intersection over union (IoU) supervision enables the model to maintain global mask accuracy while further optimizing the details of the boundary region. This design not only improves the overall robustness of the model, but also guarantees the stable performance of small-scale networks in complex scenarios.

[0055] The loss functions are only introduced in the training stage and do not add any additional parameters or computational overhead in the inference stage. Therefore, they improve the expressiveness of lightweight models without affecting the running efficiency on edge devices. In other words, by designing reasonable loss functions, the cross-modal fusion capability and boundary detection accuracy of lightweight networks are indirectly enhanced, enabling small models to approach or even reach the performance level of large models while maintaining low computational consumption, thereby further strengthening the practical application value of the invention in edge deployment scenarios such as unmanned aerial vehicle inspection, intelligent security, and vehicle auxiliary driving.

[0056] The channel consistency loss calculates adaptive weights for each channel to align the RGB and thermal imaging features: ; wherein, is learned through two fully connected layers, represents the total number of channels corresponding to the current RGB and thermal imaging features.

[0057] The deep saliency supervision loss combines binary cross-entropy and intersection over union , represented as: ; wherein, represents the true label of the layer.

[0058] The total loss is defined as: .

[0059] In summary, the application not only realizes the deep fusion of cross-modal features through structural design in the overall technical scheme, but also optimizes the lightweight and edge adaptation in a systematic way. First, a cross-modal triple attention fusion mechanism (TCM) is introduced at the feature mining level. Low-impact operators such as multi-scale dilated convolution, point-by-point convolution, and discrete cosine transform (DCT) are used to model spatial context, channel dependency, and frequency semantics, effectively replacing traditional large-scale convolution and complex multi-head attention structure. This not only ensures the capture of cross-modal details and complementary information, but also significantly reduces the parameter quantity and computational complexity. Second, a local-global complementary aggregation module (LGCA) is designed at the feature interaction level. The enhancement of local details is efficiently completed through shallow convolution and bilinear upsampling, and the global semantic modeling is realized by using global average pooling and lightweight fully connected layer. Compared with the global attention based on Transformer, the operation overhead and memory occupation are greatly reduced, while the consistency and discriminability of cross-level information are maintained. Finally, at the training optimization level, the channel consistency loss and the deep saliency supervision loss proposed by the application can effectively compensate for the insufficient feature alignment and weakened shallow expression caused by the limited capacity of the lightweight network, so that the small-scale network still has strong cross-modal fusion ability and boundary description ability without increasing the inference overhead. The synergistic effect of the three makes the overall network of the application contain only about 8.39M parameters, while maintaining high-precision detection performance and achieving near-real-time inference speed, and has the ability to run efficiently and stably on edge devices such as mobile terminals, embedded AI chips and unmanned aerial vehicles, thereby providing a lightweight and deployable solution for the practical application of salient object detection.

[0060] Based on the above embodiment, the SCF-Net of the application is evaluated on VT821, VT1000 and VT5000 and compared with 8 RGB-T salient object detection algorithms, including CPD, SGDL, MMCI, FMCF, ADF, MIDD, ECFFNet and CSRNet. The evaluation indicators include , , and . The data augmentation used in the experiment includes flipping, rotation and scale jitter, the training uses Adam learning rate, batch size is 4, and synchronous BatchNorm is performed on RTX3090.

[0061] Table 1 shows the quantitative performance of the SCF-Net of the application and other methods, in which the SCF-Net obtains 9 firsts and 1 second in 12 indicators. This confirms its robustness and accuracy, and the results highlight the advantages of the spatio-temporal-frequency fusion of the application under lightweight design.

[0062] Table 1 Quantitative comparison on VT821, VT1000 and VT5000 datasets

[0063] As shown in Table 2, compared with CPD, ADF, MIDD, CSRNet, SCF-Net achieves the lowest MAE with only 8.39M parameters and 9.13 GFLOPs, which is significantly superior to previous methods in both accuracy and efficiency. Compared with CSRNet, it reduces 13% of the computational complexity while improving the performance, demonstrating a better balance between precision and real-time capability.

[0064] Table 2 Model complexity and performance comparison Methods CPD ADF MIDD CSRNet Ours MAE 0.046 0.048 0.043 0.044 0.035 Params(M) 29.233 83.129 52.428 1.011 8.389 FLOPs(G) 118.765 382.907 433.433 10.545 9.132

[0065] The lightweight RGB-T salient object detection method supporting edge devices has obvious advantages in cross-modal feature fusion, model lightweight and practical application adaptability. Firstly, in the aspect of cross-modal feature fusion, the triple clue mining (TCM) module designed by the application jointly models the spatial structure, channel dependence and frequency semantics, which can not only extract fine-grained information within the modal, but also fully capture the deep complementary relationship between the RGB and thermal infrared modal. Cooperating with the local-global complementary aggregation (LGCA) module, the consistency of local texture and global semantics is effectively enhanced, realizing efficient information interaction across layers and modal. Compared with existing methods, the application can generate more complete and clear boundary salient object masks, and has stronger robustness and generalization ability in complex background, low illumination and camouflage environment. Secondly, in the aspects of lightweight and edge adaptation, the application takes MobileNetV2 as the backbone network, and greatly reduces the calculation amount and storage demand through depth separable convolution and reverse residual structure. The whole network contains only about 8.39M parameters, and the calculation cost is much lower than that of the mainstream ResNet or VGG backbone RGB-T detection method. Experiments show that the application realizes near real-time inference speed while ensuring high precision, which is suitable for training and inference on GPU platform, and also has the potential to run on embedded AI chips (such as NVIDIA Jetson, Huawei Ascend Atlas) and mobile terminals. Compared with existing high-energy consumption models, the application significantly reduces the computing power and energy consumption pressure of edge devices, and improves the practicability in edge scenarios such as unmanned aerial vehicle inspection, intelligent security, vehicle auxiliary driving and Internet of Things monitoring. Finally, in terms of comprehensive performance, the application not only outperforms mainstream methods on public RGB-T datasets such as VT821, VT1000 and VT5000, achieving the best result in 9 out of 12 indicators, but also maintains high inference efficiency. The application balances the precision, speed and resource occupation, proves that the method can guarantee the detection quality and meet the needs of edge deployment in practical applications, and thus has high technical value and promotion prospect.

[0066] The lightweight RGB-T salient object detection method supporting edge devices has obvious advantages in cross-modal feature fusion, model lightweight and practical application adaptability. In the aspect of cross-modal feature fusion, the triple clue mining (TCM) module designed in the application jointly models the spatial structure, channel dependence and frequency semantics, which can not only extract fine-grained information within the modal, but also fully capture the deep complementary relationship between the RGB and thermal infrared modal, and through the discrete cosine transform to decompose the high and low frequency subbands, effectively enhance the boundary definition and structural integrity of the salient object; cooperate with the local-global complementary aggregation (LGCA) module, effectively enhance the consistency of local texture and global semantics, realize efficient information interaction across levels and modalities. Compared with existing methods, the application can generate more complete and clear boundary salient object mask, and has stronger robustness and generalization ability in complex background, low illumination and camouflage environment. Secondly, in the aspects of lightweight and edge adaptation, the application takes MobileNetV2 as the backbone network, and greatly reduces the calculation amount and storage demand through depth separable convolution and reverse residual structure; the whole network contains only about 8.39M parameters, and the calculation cost is much lower than that of the mainstream ResNet or VGG backbone RGB-T detection method. Experiments show that the application realizes near real-time inference speed while ensuring high precision, which is suitable for training and inference on GPU platform, and also has the potential to run on embedded AI chips (such as NVIDIA Jetson, Huawei Ascend Atlas) and mobile terminals. Compared with existing high-energy consumption models, the application significantly reduces the computing power and energy consumption pressure of edge devices, and improves the practicability in edge scenarios such as unmanned aerial vehicle inspection, intelligent security, vehicle auxiliary driving and Internet of Things monitoring. The application balances the precision, speed and resource occupation, proves that the method of the application can guarantee the detection quality and meet the needs of edge deployment in practical application, so as to have high technical value and popularization prospect.

[0067] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Moreover, the application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) having computer-usable program code contained therein.

[0068] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0069] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0070] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks Figure 1 one or more flow or blocks

[0071] Obviously, the above-described embodiments are only examples for clarity and are not intended to limit the implementation. Based on the above description, other different forms of changes or variations can also be made by those skilled in the art. Here, it is not necessary and impossible to enumerate all the implementations. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A lightweight RGB-T salient object detection method, characterized in that, The method comprises the following steps: An RGB image to be detected and a corresponding thermal infrared image are acquired, and are respectively input into a visible light feature extraction branch and a thermal infrared feature extraction branch of an encoder of a MobileNetV2 to acquire a visible light feature and a thermal infrared feature. The first visible light feature is spliced with the second thermal infrared feature as a third multimodal fusion feature. ​​​ Let the first multi-modal fusion feature pass through the global average pooling and the fully connected layer in turn to obtain the global context feature; The global context feature, the first multi-modal fusion feature and the first visible light feature / the first thermal infrared feature are input into the first local-global complementary aggregation module in the visible light feature optimization branch / thermal infrared feature optimization branch, long-distance spatial features and the first visible light joint feature / the first thermal infrared joint feature are extracted, and are added to obtain the first visible light fusion feature / the first thermal infrared fusion feature. ​ The first The first visible light fusion feature / the first The thermal infrared fusion feature input is processed by a triple clue mining module, which extracts spatial context features, channel dependency features, and frequency clue features in parallel through spatial feature extraction, channel feature extraction, and frequency feature extraction branches. These features are then summed to obtain the first... The first visible light branching cue feature / the One thermal infrared branching cue feature; The first The first visible light branching cue feature, the first By splicing together the features of the first thermal infrared branch cue, the first... A multimodal fusion feature; The first visible feature and the second thermal infrared feature are spliced to input a first decoding unit, and a significant prediction result of the first decoding unit is obtained. ​​​​ For the first Upsample the significant prediction result of the decoding unit to obtain the first... The first upsampling result; the first The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result is input as the first... The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit; Obtaining a significant prediction result of a first decoding unit as a significant target detection result of a to-be-detected RGB image.

2. The lightweight RGB-T salient object detection method according to claim 1, wherein, An RGB image to be detected and a corresponding thermal infrared image are acquired, and are respectively input into a visible light feature extraction branch and a thermal infrared feature extraction branch of an encoder of a MobileNetV2 to acquire a visible light feature and a thermal infrared feature, including: An RGB image to be detected is acquired and its corresponding thermal infrared image , are respectively input into a visible light feature extraction branch and a thermal infrared feature extraction branch of an encoder of MobileNetV2, to acquire a first visible light feature and a first thermal infrared feature output by a first convolutional sampling unit in the visible light feature extraction branch and the thermal infrared feature extraction branch, and are represented as: ； ； The first Visible light characteristics With the thermal infrared features Input the first feature from the visible light feature extraction branch and the thermal infrared feature extraction branch respectively. In the nth convolutional sampling unit, the th... Visible light characteristics With the thermal infrared features , is represented as: ； ； wherein, , represents the total number of encoding units in the visible light feature extraction branch / thermal infrared feature extraction branch in the encoder in MobileNetV2, represents a depth separable convolution, represents down-sampling; visible light feature , thermal infrared feature , , and respectively represent the height, width and channel of the RGB image to be detected.

3. The lightweight RGB-T salient object detection method according to claim 2, wherein, A local-global complementary aggregation module comprises: A local complementary aggregation unit takes global context features as input, sequentially passes through a self-attention mechanism, upsampling and convolution, and outputs long-distance spatial features. Global complementary polymerization units, the first multi-modal fusion features, the first visible light features / the first thermal infrared features as input, let the first visible light features / the first thermal infrared features pass through self-attention mechanism and convolution, and add the first multi-modal fusion features after up-sampling and convolution, output the first visible light joint features / the first thermal infrared joint features; The fusion unit combines the input long-range spatial features, the first... The first visible light joint feature / the first After adding the combined thermal infrared features, the output is the first... The first visible light fusion feature / the first A thermal infrared fusion feature.

4. The lightweight RGB-T salient object detection method according to claim 3, characterized in that, Let the first multi-modal fusion feature pass through the global average pooling and the fully connected layer in turn to obtain the global context feature; A global complementary aggregation unit takes global context features as input, sequentially passes through a self-attention mechanism, upsampling and convolution, and outputs long-distance spatial features, comprising: ;​​​ The global feature vector of each channel is nonlinearly transformed by using a full connection layer to obtain a corresponding channel attention weight ; The first multi-modal fusion feature is weighted and adjusted by using a channel attention weight to obtain a weighted feature. ​​ concatenate the weighted features in all channels to obtain global context features ; Applying a self-attention mechanism to global context features to obtain long-range spatial features ; in, Indicates the first The first multimodal fusion feature Each channel is located in eigenvalues ​​at that location , Indicates the first The total number of channels in a multimodal fusion feature; and Indicates learnable parameters, In the weighted feature graph, the first... Each channel is located in eigenvalues ​​at that location express convolution, Indicates upsampling, This represents the self-attention mechanism.

5. The lightweight RGB-T salient object detection method according to claim 4, wherein, With the first multi-modal fusion feature, the first visible light feature / the first thermal infrared feature as input, the first visible light feature / the first thermal infrared feature is made to pass through a self-attention mechanism and convolution, and is added to the first multi-modal fusion feature after up-sampling and convolution, to output the first visible light joint feature / the first thermal infrared joint feature, comprising: Splice the first visible light branch clue feature with the first thermal infrared branch clue feature to obtain a first multimodal fusion feature ;​​​​ First visible light combined features are represented as: ； First thermal infrared joint feature is represented as: ； wherein, represents the th multimodal fusion feature; represents a concatenation operation.

6. The lightweight RGB-T salient object detection method according to claim 5, wherein, The first visible light fusion feature / the first thermal infrared fusion feature is input into a triple clue mining module, and is processed in parallel through a spatial feature extraction branch, a channel feature extraction branch and a frequency feature extraction branch. The first visible light fusion feature / the first thermal infrared fusion feature is input into a triple clue mining module, and is processed in parallel through a spatial feature extraction branch, a channel feature extraction branch and a frequency feature extraction branch. After spatial context features, channel dependency relationship features and frequency clue features are obtained, the spatial context features, the channel dependency relationship features and the frequency clue features are added to obtain the first visible light branch clue feature / the first thermal infrared branch clue feature. The first visible light fusion feature / the first thermal infrared fusion feature is input into a triple clue mining module, and is processed in parallel through a spatial feature extraction branch, a channel feature extraction branch and a frequency feature extraction branch. After spatial context features, channel dependency relationship features and frequency clue features are obtained, the spatial context features, the channel dependency relationship features and the frequency clue features are added to obtain the first visible light branch clue feature / the first thermal infrared branch clue feature. The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the spatial feature extraction branch of the triple clue mining module. After passing through the multi-scale fusion module, pooling, convolution and activation in sequence, they are residually connected with the output of the multi-scale fusion module to output spatial context features. The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the channel feature extraction branch of the triple clue mining module, and after sequentially passing through pooling, two layers of convolution and activation, they are combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output channel dependency features; The first The first visible light fusion feature / the first The thermal infrared fusion features are fed into the frequency feature extraction branch of the triple clue mining module. The input features are decomposed into four directional sub-bands using a two-dimensional discrete cosine transform and summed. After aggregation by a multilayer perceptron, the summation is combined with the first... The first visible light fusion feature / the first Residual connections are made using thermal infrared fusion features to output frequency cue features; The first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature / the A thermal infrared branching cue feature.

7. The lightweight RGB-T salient object detection method according to claim 6, wherein, Using the triple clue mining module to obtain the first The first visible light fusion feature and the first When considering the spatial context features, channel dependency features, and frequency cue features of thermal infrared fusion features, they all include: In the spatial feature extraction branch, the input features sequentially pass through a multi-scale fusion module , pooling , convolution and activation , and are connected in residual connection with the output of the multi-scale fusion module, outputting corresponding spatial context features, denoted as: ; In the channel feature extraction branch, the input features sequentially pass through pooling, two layers of convolution and activation, and are connected in residual connection with the first visible light fusion feature / the second thermal infrared fusion feature, and the corresponding channel-dependent relationship features are output, represented as: ;​​ In the frequency feature extraction branch, the input features are decomposed into four directional subbands by two-dimensional discrete cosine transform and summed, and then aggregated by a multi-layer perception, and the residual connection is performed with the first visible light fusion feature / the second thermal infrared fusion feature, and the corresponding frequency clue feature is output, which is represented as: ;​​ wherein, , represents visible light, represents thermal infrared; when, represents the first visible light fusion feature; when, represents the first thermal infrared fusion feature; represents summation, represents two-dimensional discrete cosine transform.

8. The lightweight RGB-T salient object detection method according to claim 7, characterized in that, The first The first visible light fusion feature / the first The spatial context features, channel dependency features, and frequency cue features of the thermal infrared fusion feature are added together to obtain the first... The first visible light branching cue feature / the The thermal infrared branching cue features are represented as follows: ； wherein, , represents visible light, represents thermal infrared; when, represents a first visible light branch cue feature; when, represents a first thermal infrared branch cue feature.

9. The lightweight RGB-T salient object detection method according to claim 8, characterized in that, For the first Upsample the significant prediction result of the decoding unit to obtain the first... The first upsampling result; the first... The visible light branching cue features and the first Add the features of the thermal infrared branch cue to obtain the first... The first fusion clue feature, and the second The first upsampling result, input the first... The decoding unit obtains the first decoding unit. Significant prediction results for each decoding unit include: The first Visible light characteristics With the thermal infrared features After splicing, enter the first Each decoding unit is sequentially convolved. Normalization With activation function , obtained the Significant prediction results for each decoding unit and to Perform upsampling to obtain the first Upsampling results ; Add the first visible light branch clue feature and the second thermal infrared branch clue feature to obtain a first fusion clue feature ​​​​ The first Features of fusion clues With the Upsampling results Enter the number together The decoding unit sequentially undergoes convolution, batch normalization, and activation function to obtain the th decoding unit. Significant prediction results for each decoding unit , is represented as: ; wherein , represents the total number of decoding units in the decoder in MobileNetV2.

10. The lightweight RGB-T salient object detection method according to claim 9, wherein, A total loss function is constructed to jointly train the encoder, the local-global complementary aggregation module, the triple clue mining module and the decoder; the total loss function is represented as: ； Channel consistency loss ; Deep saliency supervision loss ; wherein, denote the i-th visible light feature and the j-th thermal infrared feature, denote the i-th visible light feature and the j-th thermal infrared feature, denote the i-th visible light feature and the j-th thermal infrared feature, denote the i-th visible light feature and the j-th thermal infrared feature, denote the i-th decoding unit's significant prediction result, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label, denote the i-th decoding unit's corresponding ground truth label.

Images