Salient target detection method, device and system and electronic equipment

Abstract

The invention provides a saliency target detection method, device and system and electronic equipment, and belongs to the field of computer vision. The method comprises the following steps: acquiring a first visible light image and a first thermal infrared image of a to-be-detected object; inputting the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first saliency target image of the to-be-detected object; the first model is an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the training object; the first model comprises a self-adaptive enhancement module, a coding and fusion module and a three-stream differential cooperative decoder. In conclusion, the technical scheme provided by the invention can progressively solve the technical problems of low detection precision, weak anti-interference capability, poor fusion effect and the like of the existing method layer by layer from three core links of input enhancement, feature fusion and decoding collaboration, improves the precision of saliency target detection, and can adapt to various application scenes.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more particularly to a method, apparatus, system, and electronic device for salient object detection. Background Technology

[0002] Saliency object detection aims to mimic the human visual attention mechanism, quickly locating and segmenting the most attractive visual objects from complex scenes. As an important preprocessing step, it has been widely applied in downstream tasks such as image segmentation, object tracking, content-aware image editing, and robot navigation, making high-precision salient object detection methods essential.

[0003] However, most current salient target detection methods rely on single visible light or thermal infrared detection, or simple visible light or thermal infrared fusion detection. Both have limitations: First, single visible light detection is susceptible to changes in illumination and background clutter, making it difficult to extract effective features in low-light or shadowy environments, leading to decreased detection accuracy. Single thermal infrared detection suffers from low resolution, blurred target boundaries, and thermal cross-interference, further hindering accuracy improvement. Simultaneously, simple visible light or thermal infrared fusion detection often employs coarse fusion methods such as channel stitching and element addition, potentially ignoring the essential differences between the two modalities and failing to fully leverage their complementary advantages. This can amplify modal noise, resulting in poor fusion performance and decreased detection accuracy. In summary, existing salient target detection methods suffer from decreased detection accuracy and are ill-suited for diverse application scenarios. Summary of the Invention

[0004] This disclosure provides a method, apparatus, system, and electronic device for salient target detection, which to some extent solves the problem of decreased detection accuracy and difficulty in adapting to various application scenarios of existing salient target detection methods.

[0005] According to one aspect of this disclosure, a method for salient target detection is provided, the method comprising: acquiring a first visible light image and a first thermal infrared image of an object to be detected; inputting the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected; the first model being an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the trained object; the first model comprising: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder; the source adaptive enhancement module being used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain an enhanced third visible light image and a third thermal infrared image; the encoding and fusion module being used to extract the third visible light image respectively. The image and the third thermal infrared image are multi-scale features and cross-modal feature fusion are performed to obtain fused features; a three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second saliency target image; the three parallel decoding paths include: visible light differential decoding path, thermal infrared differential decoding path and co-fusion decoding path; the visible light differential decoding path decodes based on the third visible light image and fused features to obtain visible light differential saliency prediction features; the thermal infrared differential decoding path decodes based on the features of the third thermal infrared image and fused features to obtain thermal infrared differential saliency prediction features; the co-fusion decoding path decodes based on the visible light differential saliency prediction features, thermal infrared differential saliency prediction features and fused features to obtain the second saliency target image.

[0006] Furthermore, according to one aspect of the method of this disclosure, the source adaptive enhancement module includes: a context-aware saliency reweighting unit and an inverse gating structure unit; the context-aware saliency reweighting unit is used to receive a second visible light image and a second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; perform overlapping weighted enhancement on the second visible light image using the guiding features and cooperative gating signals to obtain a third visible light image; the inverse gating structure unit is used to receive and extract gradient features from the second visible light image and the second thermal infrared image respectively; generate inverse gating weights and structural texture features based on the visible light gradient features and the thermal infrared gradient features; perform weighted filtering on the structural texture features based on the inverse gating weights, and project the filtered structural texture features onto the second thermal infrared image for repair enhancement to obtain the third thermal infrared image.

[0007] Furthermore, according to one aspect of the method of this disclosure, the context-aware saliency reweighting unit includes: a local contrast enhancer, a guiding signal processor, a cooperative gating generator, a first element-wise multiplication layer, and a first adder; wherein, the local contrast enhancer is used to perform contrast processing on a second thermal infrared image to generate a thermal saliency map; the guiding signal processor is used to perform a first convolution and a first normalization process on the thermal saliency map to output guiding features; the cooperative gating generator is used to stitch together a second visible light image and the thermal saliency map, and after performing a second convolution, a second normalization, and a first activation process, generate a cooperative gating signal; the first element-wise multiplication layer is used to multiply the guiding features with the cooperative gating signal to obtain an enhancement term; and the first adder is used to scale the enhancement term and superimpose it with the second visible light image to obtain a third visible light image.

[0008] Furthermore, according to one aspect of the method of this disclosure, the inverse gated structure unit includes: a gradient extraction module, an inverse gated generator, a structure projector, a second element-wise multiplication layer, and a second adder; The gradient extraction module extracts gradient features from the second visible light image and the second thermal infrared image, respectively, to obtain visible light gradient features and thermal infrared gradient features. The inverse gating generator concatenates the visible light gradient features, thermal infrared gradient features, and the difference between the visible light gradient features and thermal infrared gradient features, and generates inverse gating weights through third convolution, third normalization, and second activation processing. The structure projector performs fourth convolution and fourth normalization processing on the visible light gradient features to output structure texture features. The second element-wise multiplication layer multiplies the inverse gating weights and structure texture features to obtain weighted structure texture features. The second adder scales the weighted structure texture features and superimposes them on the second thermal infrared image to obtain the third thermal infrared image.

[0009] Furthermore, according to one aspect of the method of this disclosure, the encoding and fusion module includes: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have identical structures and each includes at least five lightweight visual converter layers; for each lightweight visual converter layer, the two MobileViT-XS extract multi-scale features of a third visible light image and a third thermal infrared image, respectively, and output corresponding layer feature maps of the image; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features, and deep semantic features; the fusion unit is used to perform channel concatenation processing on the third visible light image features and the third thermal infrared image features of the same layer to obtain fused features.

[0010] Furthermore, according to one aspect of the method of this disclosure, the visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential collaborative decoder have the same structure, both including: a bilinear interpolation upsampling unit, a channel stitching unit, and a depthwise separable convolutional normalization activation unit; wherein, the bilinear interpolation upsampling unit is used to perform upsampling processing on the third visible light image or the third thermal infrared image respectively, to sequentially obtain a first upsampling feature or a second upsampling feature; the channel stitching unit is used to perform channel stitching on the first upsampling feature and the third visible light image and the fused feature, or the second upsampling feature and the third thermal infrared image and the fused feature, to obtain a first stitched feature or a second stitched feature respectively; the depthwise separable convolutional normalization activation unit is used to perform a fifth convolution, a fifth normalization, and a third activation processing on the first stitched feature or the second stitched feature, to correspondingly obtain a visible light differential saliency prediction feature or a thermal infrared differential saliency prediction feature.

[0011] Furthermore, according to one aspect of the method of this disclosure, the collaborative fusion decoding path in the three-stream differential collaborative decoder includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module; the context-aware decoupling and aggregation module is used to receive and determine a first high-level collaborative feature based on visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fusion features; the modality-aware dynamic aggregation module is used to receive and determine a second saliency target image based on the first high-level collaborative feature, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features.

[0012] Furthermore, according to one aspect of the method disclosed herein, the context-aware decoupling and aggregation module includes: a boundary attention branch, a region attention branch, a boundary enhancer, a region enhancer, a gated interaction recombination unit, and a residual connection unit; wherein, the boundary attention branch is used to perform boundary region awareness on the input features, generate a boundary attention map, and perform boundary weighting on the input features based on the boundary attention map to obtain a first boundary feature; the region attention branch is used to perform region semantic awareness on the input features, generate a region attention map, and perform region weighting on the input features based on the region attention map to obtain a first region feature; the boundary enhancer is used to perform directional enhancement on the first boundary feature to obtain a second boundary feature; the region enhancer is used to perform semantic enhancement on the first region feature to obtain a second region feature; the gated interaction recombination unit is used to perform gated weighting interaction on the second boundary feature and the second region feature to obtain a recombined feature; the residual connection unit is used to superimpose the recombined feature and the input feature to obtain a first high-level collaborative feature; the input feature is a fused feature or a spliced ​​feature; the spliced ​​feature is determined based on the recombined fused feature.

[0013] Furthermore, according to one aspect of the method disclosed herein, the modality-aware dynamic aggregation module includes: an initial fusion unit, a spatial weight generation unit, a detail enhancement branch, a structure enhancement branch, and a weighted summation unit; wherein, the initial fusion unit is used to add the first high-level collaborative feature, the visible light differential saliency prediction feature, and the thermal infrared differential saliency prediction feature element-wise to obtain the second high-level collaborative feature; the spatial weight generation unit is used to perform a sixth convolution, a sixth normalization, and a fourth activation processing on the second high-level collaborative feature to generate a spatial weight map; the detail enhancement branch is used to extract the texture information of the second high-level collaborative processing to obtain texture features; the structure enhancement branch is used to superimpose the first high-level collaborative feature and the second high-level collaborative feature to obtain structural features; and the weighted summation unit is used to perform weighted summation on the texture features and structural features using the spatial weight map to obtain the second saliency target image.

[0014] According to another aspect of this disclosure, an apparatus for salient target detection is provided. The apparatus includes: an acquisition unit for acquiring a first visible light image and a first thermal infrared image of an object to be detected; an input unit for inputting the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected; the first model is an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the trained object; the first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder; the source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain an enhanced third visible light image and a third thermal infrared image; the encoding and fusion module is used to respectively... Multi-scale features are extracted from the third visible light image and the third thermal infrared image, and cross-modal feature fusion is performed to obtain fused features. A three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second salient target image. The three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The visible light differential decoding path decodes based on the third visible light image and the fused features to obtain visible light differential saliency prediction features. The thermal infrared differential decoding path decodes based on the features of the third thermal infrared image and the fused features to obtain thermal infrared differential saliency prediction features. The co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fused features to obtain the second salient target image.

[0015] Furthermore, according to another aspect of the apparatus of this disclosure, the source adaptive enhancement module includes: a context-aware saliency reweighting unit and a reverse gating structure unit; the context-aware saliency reweighting unit is used to receive a second visible light image and a second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; perform overlapping weighted enhancement on the second visible light image using the guiding features and cooperative gating signals to obtain a third visible light image; the reverse gating structure unit is used to receive and extract gradient features from the second visible light image and the second thermal infrared image respectively; generate reverse gating weights and structural texture features based on the visible light gradient features and the thermal infrared gradient features; perform weighted filtering on the structural texture features based on the reverse gating weights, and project the filtered structural texture features onto the second thermal infrared image for repair enhancement to obtain the third thermal infrared image.

[0016] Furthermore, according to another aspect of the apparatus of this disclosure, the encoding and fusion module includes: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have identical structures and each includes at least five lightweight visual converter layers; for each lightweight visual converter layer, the two MobileViT-XS extract multi-scale features of a third visible light image and a third thermal infrared image respectively, and output the corresponding layer feature map of the image; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; the fusion units are used to perform channel splicing processing on the third visible light image features and the third thermal infrared image features of the same layer to obtain fused features.

[0017] Furthermore, according to another aspect of the apparatus of this disclosure, the visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential collaborative decoder have the same structure, both including: a bilinear interpolation upsampling unit, a channel stitching unit, and a depthwise separable convolutional normalization activation unit; wherein, the bilinear interpolation upsampling unit is used to perform upsampling processing on the third visible light image or the third thermal infrared image respectively, to sequentially obtain a first upsampling feature or a second upsampling feature; the channel stitching unit is used to perform channel stitching on the first upsampling feature and the third visible light image and the fused feature, or the second upsampling feature and the third thermal infrared image and the fused feature, to obtain a first stitched feature or a second stitched feature respectively; the depthwise separable convolutional normalization activation unit is used to perform a fifth convolution, a fifth normalization, and a third activation processing on the first stitched feature or the second stitched feature, to correspondingly obtain a visible light differential saliency prediction feature or a thermal infrared differential saliency prediction feature.

[0018] Furthermore, according to another aspect of the apparatus of this disclosure, the collaborative fusion decoding path in the three-stream differential collaborative decoder includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module; the context-aware decoupling and aggregation module is used to receive and determine a first high-level collaborative feature based on visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fusion features; the modality-aware dynamic aggregation module is used to receive and determine a second saliency target image based on the first high-level collaborative feature, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features.

[0019] According to another aspect of this disclosure, a system for salient object detection is provided. The system includes: a source adaptive enhancement module, used to receive a first visible light image and a first thermal infrared image of an object to be detected, and to perform complementary enhancement on the first visible light image and the first thermal infrared image to obtain enhanced fourth visible light image and fourth thermal infrared image; an encoding and fusion module, used to extract multi-scale features from the fourth visible light image and the fourth thermal infrared image respectively, and to perform cross-modal feature fusion to obtain fused features of the object to be detected; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features, and deep semantic features; and a three-stream differential collaborative decoder, used to decode the fourth visible light image and the fourth thermal infrared image through three parallel decoding paths. The fusion features of the thermal infrared image and the object to be detected are decoded to obtain the first salient target image. The three parallel decoding paths include: visible light differential decoding path, thermal infrared differential decoding path, and collaborative fusion decoding path. The visible light differential decoding path decodes based on the fusion features of the fourth visible light image and the object to be detected to obtain the visible light differential saliency prediction features of the object to be detected. The thermal infrared differential decoding path decodes based on the features of the fourth thermal infrared image and the fusion features of the object to be detected to obtain the thermal infrared differential saliency prediction features of the object to be detected. The collaborative fusion decoding path decodes based on the visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fusion features of the object to be detected to obtain the first salient target image.

[0020] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing computer-readable instructions; and a processor for executing the computer-readable instructions, causing the electronic device to perform the method as described in any embodiment of one aspect.

[0021] This disclosure provides a method, apparatus, system, and electronic device for salient target detection. The disclosure involves acquiring a first visible light image and a first thermal infrared image of an object to be detected; inputting the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected; the first model is an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the trained object; the first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder; the source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain enhanced third visible light images and third thermal infrared images; the encoding and fusion module is used to extract multi-scale values ​​from the third visible light image and the third thermal infrared image respectively. Features are fused across modalities to obtain fused features; a three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second saliency target image; the three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path; the visible light differential decoding path decodes based on the third visible light image and fused features to obtain visible light differential saliency prediction features; the thermal infrared differential decoding path decodes based on the third thermal infrared image features and fused features to obtain thermal infrared differential saliency prediction features; the co-fusion decoding path decodes based on the visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fused features to obtain the second saliency target image. In this way, compared with existing single-modality detection methods and simple fusion detection methods, this disclosure can specifically address the following issues: For the problems of weak anti-interference capability of single visible light detection and blurred boundaries of single thermal infrared detection, this disclosure enhances the effective complementary enhancement of both through a source adaptive enhancement module. Specifically, it utilizes the thermal radiation characteristics of the thermal infrared image to perform overlapping weighted enhancement on the visible light image, suppressing background clutter and illumination interference. Simultaneously, it uses the high-frequency structural features of the visible light image to repair the blurred boundaries of the thermal infrared image and alleviate thermal cross-interference, thereby improving the input quality of both modalities from the source. Furthermore, for the problem of simple fusion detection ignoring modal differences and resulting in coarse fusion, this disclosure can extract multi-scale features of the two enhanced modalities separately through encoding and fusion modules, achieving accurate cross-modal feature integration. Moreover, through the three parallel paths of the three-stream differential collaborative decoder, it refines the decoding of the two modal features and the fused features, and then achieves dynamic synergy of the predicted features of the two modalities through a collaborative fusion path, further improving detection accuracy. In summary, the technical solution provided in this disclosure can address the technical problems of existing methods, such as low detection accuracy, weak anti-interference ability, and poor fusion effect, by addressing the three core aspects of input enhancement, feature fusion, and decoding collaboration in a progressive manner. This improves the accuracy of salient target detection and can be adapted to various application scenarios.

[0022] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0023] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 A schematic flowchart of a salient target detection method provided in this embodiment of the disclosure; Figure 2 A structural block diagram of the first model training and testing provided in the embodiments of this disclosure; Figure 3 A schematic diagram of the composition structure of the context-aware saliency reweighting unit provided in the embodiments of this disclosure; Figure 4 A schematic diagram of the composition structure of the reverse gating structure projection unit provided in the embodiments of this disclosure; Figure 5 A schematic diagram illustrating the composition of the visible light differential decoding path or the thermal infrared differential decoding path of the three-stream differential co-decoder provided in the embodiments of this disclosure; Figure 6 A schematic diagram illustrating the composition structure of the collaborative fusion decoding path of the three-stream differential collaborative decoder provided in this embodiment of the disclosure; Figure 7 A schematic diagram illustrating the composition structure of the context-aware decoupling and aggregation module provided in this embodiment of the disclosure; Figure 8 A schematic diagram of the composition structure of the modality sensing dynamic aggregation module provided in the embodiments of this disclosure; Figure 9 A schematic diagram of the complete composition structure of the first model provided for embodiments of this disclosure; Figure 10 A structural block diagram of a salient target detection device provided in this disclosure embodiment; Figure 11 This is a hardware block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0026] Current methods for salient target detection mostly rely on single visible light or thermal infrared detection, or simple visible light or thermal infrared fusion detection. Both have limitations: First, single visible light detection is susceptible to changes in illumination and cluttered backgrounds, making it difficult to extract effective features in low-light or shadowy environments, leading to decreased detection accuracy. Single thermal infrared detection suffers from low resolution, blurred target boundaries, and thermal cross-interference, further hindering accuracy improvement. Second, simple visible light or thermal infrared fusion detection often employs coarse fusion methods such as channel stitching and element addition, potentially ignoring the essential differences between the two modalities and failing to fully leverage their complementary advantages. This can amplify modal noise, resulting in poor fusion performance and decreased detection accuracy. In summary, existing salient target detection methods suffer from decreased detection accuracy and are ill-suited for diverse application scenarios.

[0027] Therefore, in response to the aforementioned problems, this disclosure provides a method for salient target detection. Compared to existing single-modal detection methods and simple fusion detection methods, this disclosure offers targeted solutions: For the issues of weak anti-interference capability of single visible light detection and blurred boundaries of single thermal infrared detection, this disclosure enhances the effective complementary enhancement of both through a source adaptive enhancement module. Specifically, it utilizes the thermal radiation characteristics of the thermal infrared image to perform overlapping weighted enhancement on the visible light image, suppressing background clutter and illumination interference. Simultaneously, it utilizes the high-frequency structural features of the visible light image to repair the blurred boundaries of the thermal infrared image and alleviate thermal cross-interference, thereby improving the input quality of both modalities from the source. Furthermore, addressing the problem of simple fusion detection ignoring modal differences and resulting in coarse fusion, this disclosure extracts multi-scale features of the two enhanced modalities separately through encoding and fusion modules, achieving accurate cross-modal feature integration. Moreover, through the three parallel paths of the three-stream differential collaborative decoder, it performs refined decoding of the two modal features and the fused features, and then achieves dynamic synergy of the predicted features of the two modalities through a collaborative fusion path, further improving detection accuracy. In summary, the technical solution provided in this disclosure can address the technical problems of existing methods, such as low detection accuracy, weak anti-interference ability, and poor fusion effect, by addressing the three core aspects of input enhancement, feature fusion, and decoding collaboration in a progressive manner. This improves the accuracy of salient target detection and can be adapted to various application scenarios.

[0028] First, please refer to Figure 1 , Figure 1 This is a schematic flowchart illustrating a salient target detection method provided in an embodiment of this disclosure. Figure 1As shown, the method includes: In step S101, a first visible light image and a first thermal infrared image of the object to be detected are acquired; In step S102, the first visible light image and the first thermal infrared image are input into the pre-trained first model to obtain the first salient target image of the object to be detected. The first model is an image fusion neural network model determined based on the second visible light image and the second thermal infrared image of the trained object. The first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder. The source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain the enhanced third visible light image and the third thermal infrared image. The encoding and fusion module is used to extract multi-scale features from the third visible light image and the third thermal infrared image respectively and perform cross-modal feature extraction. The fusion process yields fused features; a three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second saliency target image; the three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path; the visible light differential decoding path decodes based on the third visible light image and the fused features to obtain visible light differential saliency prediction features; the thermal infrared differential decoding path decodes based on the third thermal infrared image features and the fused features to obtain thermal infrared differential saliency prediction features; the co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fused features to obtain the second saliency target image.

[0029] In this disclosure, the first model can be understood as a lightweight visible light and thermal infrared dual-modal salient target detection neural network model for complex environments. It learns complementary features of visible light and thermal infrared modes during the training phase and can perform end-to-end salient target detection on the input image during the testing phase, improving detection accuracy in complex scenes while ensuring lightweight design.

[0030] Specifically, the first model disclosed herein includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder.

[0031] Among them, the source adaptive enhancement module can be understood as a preprocessing module located at the model input end, which is used to perform targeted enhancement on the original visible light and thermal infrared images before feature extraction: on the one hand, it uses thermal infrared priors to suppress background clutter and illumination interference in visible light, and on the other hand, it uses the high-frequency structure of visible light to repair the blurred boundaries and thermal cross-interference of thermal infrared, thereby improving the input quality of the two modes from the source.

[0032] The encoding and fusion module can be understood as a lightweight multi-scale feature extraction and cross-modal integration module. It can use two structurally consistent lightweight visual transformer networks (MobileVisionTransformer-ExtraSmall, MobileViT-XS) backbone networks to extract multi-scale features from the enhanced visible light and thermal infrared images, respectively. Then, the lightweight fusion unit performs channel splicing and convolution fusion on the two modal features at the same level to obtain multi-scale fused features that take into account the information of both modalities, avoiding the noise amplification problem caused by coarse fusion, which will be elaborated later.

[0033] The three-stream differential co-decoder can be understood as a refined decoding module containing three parallel paths: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The first two paths independently decode the visible light and thermal infrared features to obtain their respective saliency prediction features. The co-fusion path, based on the outputs and fused features of the first two paths, achieves dynamic coordination of cross-modal prediction results and ultimately outputs a complete saliency target mask, solving the problem that a single decoding path cannot adapt to the spatial heterogeneity of modal reliability.

[0034] Specifically, when using the first model for salient object detection, it includes: The first visible light image and the first thermal infrared image to be detected are input into the source adaptive enhancement module to obtain the enhanced visible light image and thermal infrared image, respectively. The two enhanced images are then input into the encoding and fusion module to extract multi-scale visible light features and thermal infrared features, and cross-modal fusion is performed to obtain fused features. The visible light features, thermal infrared features, and fused features are input into the three-stream differential collaborative decoder. The corresponding saliency prediction features are obtained through the visible light differential decoding path and the thermal infrared differential decoding path, respectively. Then, the two prediction features and the fused features are dynamically and collaboratively decoded through the collaborative fusion decoding path to obtain the final first saliency target image output.

[0035] For example, Figure 2 A structural block diagram illustrating the first model training and testing provided in an embodiment of this disclosure. Figure 2 It can be seen that the first model can be divided into two stages: The first stage is the training phase: the training set data is first processed by the source adaptive enhancement module to obtain enhanced visible light and thermal infrared images, which are then input into the lightweight encoding and fusion module, outputting multi-scale visible light feature maps, multi-scale thermal infrared feature maps, and multi-scale fused feature maps. Subsequently, the visible light differential decoding path, thermal infrared differential decoding path, and collaborative fusion decoding path are input respectively to obtain their respective saliency prediction feature maps. The outputs of each path are compared with the real images to calculate the loss function, and the model parameters are optimized through backpropagation to complete model training. For example, the loss function of this disclosure can satisfy the following:

[0036] Where L represents the total loss function; Represented as the first The saliency prediction feature map of the decoded path output; This is expressed as a binary cross-entropy loss; This is expressed as the intersection-union loss; Represented as a true salient target image.

[0037] It should be noted that the loss function described above is only a preferred example, and the loss function can be flexibly changed according to the application scenario without any restrictions.

[0038] The second stage, the testing stage, follows the same process as the training stage, except that the loss function calculation step is omitted. The collaborative saliency prediction feature map output from the collaborative fusion decoding path is directly used as the final saliency target image output, achieving efficient end-to-end detection.

[0039] The following will elaborate on the source adaptive enhancement module in the first module of this disclosure, including: a context-aware saliency reweighting unit and a reverse gating structure unit.

[0040] First, let's explain the significantly reweighted unit of up-down perception: The context-aware saliency reweighting unit is used to receive a second visible light image and a second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; and perform overlapping weighted enhancement on the second visible light image using the guiding features and cooperative gating signals to obtain a third visible light image. Specifically, the context-aware saliency reweighting unit includes: a local contrast enhancer, a guiding signal processor, a cooperative gating generator, a first element-wise multiplication layer, and a first adder; The local contrast enhancer is used to perform contrast processing on the second thermal infrared image to generate a thermal saliency map; the guiding signal processor is used to perform a first convolution and a first normalization on the thermal saliency map to output guiding features; the cooperative gating generator is used to stitch together the second visible light image and the thermal saliency map, and after a second convolution, a second normalization and a first activation, generate a cooperative gating signal; the first element-wise multiplication layer is used to multiply the guiding features with the cooperative gating signal to obtain an enhancement term; and the first adder is used to scale the enhancement term and superimpose it with the second visible light image to obtain a third visible light image.

[0041] In this disclosure, the context-aware saliency reweighting unit can be understood as a preprocessing subunit in the source adaptive enhancement module that specifically processes visible light images. It uses the thermal radiation information of the thermal infrared image as a priori to perform adaptive reweighting enhancement on the visible light image.

[0042] The context-aware saliency reweighting unit disclosed herein includes: a local contrast enhancer, a guiding signal processor, a cooperative gating generator, a first element-wise multiplication layer, and a first adder. The local contrast enhancer can be understood as consisting of an average pooling layer, a subtractor, and an activation function, used to calculate the local contrast of the second thermal infrared image, generate a thermal saliency map, and locate the target region in the thermal infrared image. The guiding signal processor can be understood as consisting of two 1×1 convolutional layers, a batch normalization layer, and an activation function, used to extract and normalize features from the thermal saliency map, generate guiding features, and clarify the location of the target region to be enhanced. The cooperative gating generator can be understood as consisting of a channel stitching layer, a 3×3 convolutional layer, a batch normalization layer, an activation function, a 1×1 convolutional layer, and a logistic function, used to stitch the second visible light image and the thermal saliency map, learn the modal correlation between the two, generate a cooperative gating signal, and achieve precise control of the enhanced region. The first element-wise multiplication layer can be understood as multiplying the guiding features and the cooperative gating signal element-wise to obtain a precise enhancement term, which enhances only the target region and avoids amplifying background noise. The first adder can be understood as weighting the enhancement term by a scaling factor and then adding it element-wise to the second visible light image, finally outputting the enhanced third visible light image.

[0043] For example, Figure 3 This is a schematic diagram illustrating the composition of a context-aware saliency reweighting unit provided in an embodiment of this disclosure. Specifically, the following will be combined with... Figure 3 This section details how the up-and-down perceptual saliency-weighted unit performs visible light image enhancement, including: First, the second thermal infrared image The input local contrast enhancer is then processed by an average pooling layer to obtain a local mean map. Then calculate using a subtractor. and The difference is used to obtain a heatsaliency map after activation by an activation function. Locate the target area in the thermal infrared image.

[0044] Next, the heat map will be displayed. The input signal processor processes the signal through 1×1 convolution, batch normalization, activation function activation, and then another 1×1 convolution to obtain the guiding features. Clearly define the characteristics of the target areas that need to be enhanced.

[0045] Then, the second visible light image With thermal saliency map The input to the cooperative gating generator is processed through channel concatenation, 3×3 convolution, batch normalization, activation function activation, and 1×1 convolution, followed by activation by the logistic function to obtain the cooperative gating signal. Control the intensity and range of the enhancement.

[0046] Furthermore, guiding features With coordinated gating signals The first element is input and multiplied in turn; the enhancement term is obtained by multiplying each element in turn. .

[0047] Finally, the enhancements Scaling factor After weighting, it is compared with the second visible light image. The input is fed into the first adder, and the enhanced third visible light image is obtained by element-wise superposition. This completes the saliency reweighting enhancement of visible light images.

[0048] Next, the reverse gating structure unit will be explained: The reverse gating structure unit is used to receive and extract the gradient features of the second visible light image and the second thermal infrared image respectively; based on the visible light gradient features and the thermal infrared gradient features, reverse gating weights and structural texture features are generated; the structural texture features are weighted and filtered based on the reverse gating weights, and the filtered structural texture features are projected onto the second thermal infrared image for repair and enhancement to obtain the third thermal infrared image.

[0049] Specifically, the inverse gated structure unit includes: a gradient extraction module, an inverse gated generator, a structure projector, a second element multiplication layer, and a second adder; The gradient extraction module extracts gradient features from the second visible light image and the second thermal infrared image, respectively, to obtain visible light gradient features and thermal infrared gradient features. The inverse gating generator concatenates the visible light gradient features, thermal infrared gradient features, and the difference between the visible light gradient features and thermal infrared gradient features, and generates inverse gating weights through third convolution, third normalization, and second activation processing. The structure projector performs fourth convolution and fourth normalization processing on the visible light gradient features to output structure texture features. The second element-wise multiplication layer multiplies the inverse gating weights and structure texture features to obtain weighted structure texture features. The second adder scales the weighted structure texture features and superimposes them on the second thermal infrared image to obtain the third thermal infrared image.

[0050] In this disclosure, the reverse gating structure unit can be understood as a preprocessing subunit in the source adaptive enhancement module that is specifically designed to process thermal infrared images. Guided by the high-frequency structural information of the visible light image, it adaptively projects clear edge and texture features onto the thermal infrared image through the reverse gating mechanism, thereby repairing defects such as blurred boundaries and thermal crossover in the thermal infrared image and improving the structural integrity and target recognition of the thermal infrared image.

[0051] The disclosed inverse gating structure unit includes: a gradient extraction module, an inverse gating generator, a structure projector, a second element-wise multiplication layer, and a second adder. The gradient extraction module can be understood as consisting of a Sobel operator and two 1×1 convolutional layers, used to extract gradient features from a second visible light image and a second thermal infrared image, respectively, to obtain visible light gradient features and thermal infrared gradient features. The inverse gating generator can be understood as consisting of a channel stitching layer, a 1×1 convolutional layer, a BatchNormalization layer, an activation function, and a logistic function, used to stitch together visible light gradient features, thermal infrared gradient features, and difference features, learn the structural differences between modes, generate inverse gating weights, and achieve precise control of the structure projection region. The structure projector can be understood as consisting of a 3×3 convolutional layer, a BatchNormalization layer, an activation function, and a 1×1 convolutional layer, used to extract and enhance visible light gradient features, outputting structural texture features. The second element-wise multiplication layer can be understood as multiplying the inverse gated weights element-wise with the structural texture features to obtain weighted structural texture features. Visible light structural information is injected only into the areas that need repair, avoiding excessive interference with the original thermal radiation information of the thermal infrared image. The second adder can be understood as weighting the weighted structural texture features by a scaling factor and then adding them element-wise to the second thermal infrared image, finally outputting the enhanced third thermal infrared image.

[0052] For example, Figure 4 This is a schematic diagram of the composition of the reverse gating structure projection unit provided in an embodiment of this disclosure. The following will be combined with... Figure 4 This section elaborates on how the reverse-gated structure projection unit enhances thermal infrared images, including: First, the second visible light image Second thermal infrared image The gradients are input into the gradient extraction module and calculated using the Sobel operator to obtain the visible light gradient map. Thermal infrared gradient map The visible light gradient features are then obtained by processing the data through a 1×1 convolutional layer. and thermal infrared gradient characteristics The difference features are obtained by subtraction. .

[0053] Next, the visible light gradient features Thermal infrared gradient characteristics and differences in characteristics The input to the inverse gating generator is processed through channel concatenation, 1×1 convolution, batch normalization, and activation. After further 1×1 convolution and logistic function activation, the inverse gating weights are obtained. It is used to control the area and intensity of the structural projection.

[0054] Then, the visible light gradient map The input structure projector is processed by 3×3 convolution, BatchNormalization, activation function, and then 1×1 convolution to obtain the structure texture feature Sp, which extracts the clear edge and texture information in the visible light image.

[0055] Furthermore, structural texture features With reverse gating weights The second element is input as the multiplication layer, and the weighted structural texture features are obtained by multiplying element by element. This enables precise selection of the repair area.

[0056] Finally, the weighted structural texture features After being weighted by scaling factor α, it is compared with the second thermal infrared image. The input is fed into the second adder, and the enhanced third thermal infrared image is obtained by element-wise superposition. This completes the boundary repair and structural enhancement of thermal infrared images.

[0057] The encoding and fusion module in the first model of this disclosure will be described in detail below, including: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have the same structure and each includes at least five lightweight visual converter layers; For each lightweight vision converter layer, the two MobileViT-XS extract multi-scale features from the third visible light image and the third thermal infrared image, respectively, and output the corresponding layer feature map of the image. The multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features. The fusion unit is used to perform channel stitching processing on the third visible light image features and the third thermal infrared image features at the same level to obtain fused features.

[0058] In this disclosure, the encoding and fusion module can be understood as a lightweight encoding module used to extract multi-scale features from enhanced visible light images and thermal infrared images and to achieve efficient cross-modal feature fusion. While ensuring feature representation capabilities, it significantly reduces the number of model parameters and computational load.

[0059] The encoding and fusion module disclosed herein includes two lightweight visual converter networks, MobileViT-XS, and at least five fusion units. The two MobileViT-XS networks are structurally identical and are used to perform layer-by-layer feature extraction on the enhanced third visible light image and the third thermal infrared image, respectively, outputting multi-scale feature maps at five corresponding levels, covering image texture, contour, local semantics, global semantics, and deep semantic features. Each lightweight visual converter layer is used to capture visual information from different receptive fields of the input features; shallow layers extract details and edge structures, while deep layers extract high-level semantics and global contextual information. The at least five fusion units and the features at each level... Figure 1 One-to-one correspondence is used to stitch together and integrate visible light features and thermal infrared features at the same level to obtain multi-scale fusion features that take into account the advantages of both modes, providing a foundation for subsequent three-stream differential collaborative decoding.

[0060] The following will elaborate on the three-stream differential collaborative decoder in the first module of this disclosure, including three paths: visible light differential decoding path, thermal infrared differential decoding path, and collaborative fusion decoding path.

[0061] First, let's explain the visible light differential decoding path and the thermal infrared differential decoding path, including: The visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential co-decoder have the same structure, both including: bilinear interpolation upsampling unit, channel splicing unit, and depthwise separable convolutional normalized activation unit; The bilinear interpolation upsampling unit is used to upsample the third visible light image or the third thermal infrared image respectively, to obtain the first upsampled feature or the second upsampled feature in sequence; the channel stitching unit is used to stitch the first upsampled feature and the third visible light image and the fused feature, or the second upsampled feature and the third thermal infrared image and the fused feature, to obtain the first stitched feature or the second stitched feature respectively; the depthwise separable convolutional normalization activation unit is used to perform the fifth convolution, the fifth normalization and the third activation processing on the first stitched feature or the second stitched feature, to obtain the visible light differential saliency prediction feature or the thermal infrared differential saliency prediction feature.

[0062] In this disclosure, the visible light differential decoding path can be understood as a sub-module that performs layer-by-layer decoding of the multi-scale features and fused features of the enhanced visible light image. Through step-by-step upsampling and feature fusion, the salient target details in the visible light mode are restored, and the visible light differential saliency prediction feature map is output.

[0063] In this disclosure, the thermal infrared differential decoding path can be understood as a sub-module that performs layer-by-layer decoding of the multi-scale features and fusion features of the enhanced thermal infrared image. Through a structure consistent with the visible light differential decoding path, it restores the salient target region in the thermal infrared mode and outputs the thermal infrared differential saliency prediction feature map.

[0064] In this disclosure, the visible light differential decoding path and the thermal infrared differential decoding path have the same structure, both including: a bilinear interpolation upsampling unit, a channel concatenation unit, and a depthwise separable convolutional normalized activation unit. The bilinear interpolation upsampling unit can be understood as upsampling the decoded output features of the upper layer or the deep features of the backbone network, aligning the feature map size with the current layer's feature map. The channel concatenation unit can be understood as concatenating the upsampled features, the current layer's modal features (visible light / thermal infrared), and the corresponding layer's fused features along the channel dimension, integrating multi-scale, cross-modal contextual information. The depthwise separable convolutional normalized activation unit can be understood as consisting of a 3×3 depthwise separable convolutional layer, a batch normalization layer, and an activation function, used for efficient feature extraction and noise reduction of the concatenated features, while simultaneously optimizing the feature dimension through pointwise convolution to output the differential features of the current layer.

[0065] For example, Figure 5 This is a schematic diagram illustrating the composition of the visible light differential decoding path or the thermal infrared differential decoding path of the three-stream differential co-decoder provided in the embodiments of this disclosure. The following will be combined with... Figure 5 The principles of decoding using visible light differential decoding paths or thermal infrared differential decoding paths are explained in detail, including: First, the bilinear interpolation upsampling layer receives the feature input from the previous layer (when i=4, the input is the modal features of the 5th layer output by the backbone network). When i < 4, the input is the differential features of the (i+1)th layer. Output upsampled features consistent with the current level size. .

[0066] Next, the channel splicing layer will upsample the features. Current level modal features and corresponding hierarchical fusion features Perform channel splicing to obtain splicing features. .

[0067] Then, a 3×3 depth-separable convolutional layer receives the spliced ​​features. After batch normalization and activation by an activation function, the output depthwise convolutional features are obtained. .

[0068] Next, the pointwise convolutional layer receives depthwise convolutional features. After batch normalization and activation by the activation function, the differential features of the current level are output. (Visible light path) or (Thermal infrared path).

[0069] Finally, repeat the above process from i=4 to i=1, and finally output the result when i=1. and This is the visible light difference significance prediction feature map. Significance prediction feature map of thermal infrared difference .

[0070] Next, the collaborative fusion decoding path is explained, including: the context-aware decoupling and aggregation module, and the modality-aware dynamic aggregation module; The context-aware decoupling and aggregation module is used to receive and determine the first high-level collaborative feature based on visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fused features; The modality-aware dynamic aggregation module is used to receive and determine the second salient target image based on the first high-level collaborative features, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features.

[0071] In this disclosure, the collaborative fusion decoding path can be understood as the core cross-modal collaborative sub-module of the three-stream differential collaborative decoder. Based on visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and multi-scale fusion features, it adaptively fuses the advantageous information of the two modes through dynamic spatial aggregation and context-aware decoupling, and finally outputs a high-precision collaborative saliency prediction feature map, solving the problem that single-modal decoding cannot adapt to spatial heterogeneity.

[0072] The collaborative fusion decoding path disclosed herein includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module. The context-aware decoupling and aggregation module can be understood as performing context modeling and decoupling on the concatenated features, separating redundant information from different modalities, aggregating effective saliency features, and outputting the collaborative decoding features of the current level; simultaneously, it receives the fusion features from the 5th level as initial input to generate top-level collaborative features. The modality-aware dynamic aggregation module can be understood as receiving high-level collaborative features, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features, adaptively aggregating the prediction information of the two modalities based on the modal reliability differences at different spatial locations, and outputting the aggregated features.

[0073] For example, Figure 6 This is a schematic diagram illustrating the composition structure of the collaborative fusion decoding path of the three-stream differential collaborative decoder provided in this embodiment of the disclosure. The following will be combined with... Figure 6 The specific explanation of how the collaborative fusion decoding path is decoded includes: First, the 5th level will only receive fused features. Input the context-aware decoupling and aggregation module to generate initial collaborative features. .

[0074] Next, bilinear interpolation upsampling is performed on the collaborative decoding output features of the (i+1)th level to obtain the fused features with the current level. Consistent size upsampling features As a feature of high-level collaboration .

[0075] Then, high-level collaborative features Current level visible light differential features Differential characteristics with thermal infrared Input dynamic spatial aggregation module, output aggregated features This enables feature fusion under modal reliability perception.

[0076] Furthermore, aggregate features Features integrated with the current level Perform channel splicing to obtain splicing features. .

[0077] Furthermore, the splicing features Input the context-aware decoupling and aggregation module, and output the collaborative decoding features of the current layer. .

[0078] Finally, repeat the above process from i=4 to i=1, and finally output the collaborative features at i=1. This is the feature map for predicting co-significance. This refers to the second salient target image.

[0079] Furthermore, the context-aware decoupling and aggregation module will be specifically described, including: boundary attention branch, region attention branch, boundary enhancer, region enhancer, gated interaction reorganization unit, and residual connection unit; The system comprises the following components: a boundary attention branch for boundary region perception of input features, generating a boundary attention map, and applying boundary weights to the input features based on the boundary attention map to obtain a first boundary feature; a region attention branch for region semantic perception of input features, generating a region attention map, and applying region weights to the input features based on the region attention map to obtain a first region feature; a boundary enhancer for directional enhancement of the first boundary feature to obtain a second boundary feature; a region enhancer for semantic enhancement of the first region feature to obtain a second region feature; a gated interactive recombination unit for gated weighted interaction of the second boundary feature and the second region feature to obtain a recombined feature; and a residual connection unit for superimposing the recombined feature and the input feature to obtain a first high-level collaborative feature. The input feature is either a fused feature or a spliced ​​feature. The spliced ​​feature is determined based on the recombined fused feature.

[0080] In this disclosure, the input features can be understood as the input features of the context-aware decoupling and aggregation module, which have two forms at different decoding stages: The first form: When the module is in the top-level decoding stage, the input features are fused features (i.e., multi-scale fused features output by the encoding and fusion modules). The second form: When the module is in the middle or bottom layer decoding stage, the input feature is the spliced ​​feature (that is, the feature obtained by splicing the aggregated feature output by the modality-aware dynamic aggregation module with the current level fusion feature through the channel).

[0081] The context-aware decoupling and aggregation module disclosed herein includes: a boundary attention branch, a region attention branch, a boundary enhancer, a region enhancer, a gated interaction reassembly unit, and a residual connection unit. The boundary attention branch can be understood as consisting of a 3×3 depthwise separable convolutional layer and a logistic function, used for boundary region awareness of input features, generating a boundary attention map, and then obtaining boundary features through element-wise multiplication, highlighting target edge information. The region attention branch can be understood as consisting of a 3×3 depthwise separable convolutional layer with an inflation rate of 2 and a logistic function, used for region semantic awareness of input features, generating a region attention map, and then obtaining region features through element-wise multiplication, strengthening the semantic information inside the target. The boundary enhancer can be understood as consisting of horizontal / vertical strip convolutional layers, adders, and pointwise convolutional layers, used for directional enhancement of boundary features, repairing edge breaks, and obtaining clearer boundary features. The region enhancer can be understood as consisting of three parallel 3×3 depthwise separable convolutional layers with different dilation rates, a channel concatenation layer, and a 1×1 convolutional layer. It is used to perform multi-scale semantic enhancement of region features, enriching the internal details of the target and obtaining more complete region features. The gated interactive recombination unit can be understood as consisting of a 3×3 depthwise separable convolutional layer, a logistic function, an element-wise multiplication layer, an adder, and a 3×3 convolutional layer. It is used to perform gated weighted interaction between the enhanced boundary and region features, achieving dynamic recombination of boundary and region information. The residual connection unit can be understood as consisting of a 3×3 convolutional layer and an adder. It is used to perform residual superposition of the recombined features and the original input features, preserving the original information and mitigating gradient vanishing, outputting the final high-level collaborative features.

[0082] For example, Figure 7 This is a schematic diagram illustrating the composition structure of the context-aware decoupling and aggregation module provided in an embodiment of this disclosure. The following will be combined with... Figure 7 Specifically, this section explains how the context-aware decoupling and aggregation module determines high-level collaborative characteristics, including: First, input features to the module (when i=1, 2, 3, 4, these are the splicing features) are selected. When i=5, it is the fusion feature. The input boundary attention branch is processed by a 3×3 depthwise separable convolution and logistic activation to obtain the boundary attention map. The boundary features are obtained by multiplying them element-wise with the module input features. ; Simultaneously, the module input features are input into the region attention branch, and after undergoing a 3×3 depthwise separable convolution with a dilation rate of 2 and logistic activation, a region attention map is obtained. The region features are obtained by multiplying them element-wise with the module input features. .

[0083] Next, The input boundary enhancer extracts orientation information through horizontal / vertical strip convolutions, which are then summed and convolved pointwise to obtain enhanced boundary features. The Xr input is used to enhance the region, which is then subjected to three parallel depthwise separable convolutions with dilation rates of 1, 2, and 3 to extract multi-scale semantics. After channel concatenation, the enhanced region features are obtained through a 1×1 convolution. .

[0084] Then, The input gated interactive reconstruction unit is processed by a 3×3 depthwise separable convolution and logistic activation to obtain the gated signal. ;Will and Element-wise multiplication yields weighted boundary features , and then with Adding them together gives Recombination features are obtained through 3×3 convolution. .

[0085] Finally, the module input features are input into a 3×3 convolutional layer of the residual connection unit to obtain the residual features. ; recombination characteristics and By adding elements one by one, the final high-level collaborative characteristics are obtained. .

[0086] Furthermore, the modality-aware dynamic aggregation module will be specifically described, including: an initial fusion unit, a spatial weight generation unit, a detail enhancement branch, a structure enhancement branch, and a weighted summation unit; The initial fusion unit is used to add the first high-level collaborative features, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features element-wise to obtain the second high-level collaborative features; the spatial weight generation unit is used to perform the sixth convolution, sixth normalization, and fourth activation processing on the second high-level collaborative features to generate a spatial weight map; the detail enhancement branch is used to extract the texture information of the second high-level collaborative processing to obtain texture features; the structure enhancement branch is used to superimpose the first high-level collaborative features and the second high-level collaborative features to obtain structural features; and the weighted summation unit is used to perform weighted summation on the texture features and structural features using the spatial weight map to obtain the second saliency target image.

[0087] The modality-aware dynamic aggregation module disclosed herein includes: an initial fusion unit, a spatial weight generation unit, a detail enhancement branch, a structure enhancement branch, and a weighted summation unit. The initial fusion unit can be understood as consisting of three 1×1 convolutional layers and an adder, used to align and element-wise add high-level collaborative features, visible light differential features, and thermal infrared differential features to obtain initial fused features, providing a foundation for subsequent modality-aware aggregation. The spatial weight generation unit can be understood as consisting of a 3×3 depthwise separable convolutional layer and a normalization function, used to spatially model the initial fused features, outputting two complementary spatial weight maps, adaptively allocating weights to texture and structural information. The detail enhancement branch can be understood as consisting of a 3×3 depthwise separable convolutional layer, a logistic function, and an element-wise multiplication layer, used to extract texture detail information from the initial fused features, generate texture features, and enhance target edges and details. The structure enhancement branch can be understood as consisting of an adder, used to element-wise add the thermal infrared differential features to the initial fused features to generate structural features, highlighting the overall structure and thermal radiation information of the target. The weighted summation unit can be understood as consisting of two element-wise multiplication layers and an adder. It is used to perform weighted summation of texture features and structural features using a spatial weight map to obtain aggregated features. Then, it is passed through a 3×3 depthwise separable convolution and residual connection to output the final aggregated features.

[0088] For example, Figure 8 This is a schematic diagram illustrating the composition of the modality-sensing dynamic aggregation module provided in an embodiment of this disclosure. The following will be combined with... Figure 8 The modality-aware dynamic aggregation module disclosed herein includes: First, the characteristics of high-level collaboration Visible light differential characteristics Thermal infrared differential characteristics Inputting each of the three 1×1 convolutional layers yields dimension-aligned features. , , The initial fusion characteristics are obtained by adding elements one by one using an adder. .

[0089] Next, The input spatial weight generation unit is activated by a 3×3 depthwise separable convolution and a normalization function, and outputs two complementary spatial weight maps α and β, which correspond to the spatial weights of texture and structure, respectively.

[0090] Then, Detail gating is obtained by 3×3 depthwise separable convolution and logistic activation. ,and Element-wise multiplication yields detailed features , and then with Adding them together yields the texture features. ;Will and By adding elements one by one, we obtain the structural features. .

[0091] Finally, texture features Element-wise multiplication with spatial weight graph α yields Structural features Element-wise multiplication with spatial weight graph β yields The weighted aggregate feature is obtained by adding the results using an adder. Then, after 3×3 depthwise separable convolution, we get ;Will Residual features are obtained through 3×3 convolution. ,and Add elements one by one to output the final aggregated feature. .

[0092] For example, Figure 9 A schematic diagram illustrating the complete structural composition of the first model provided for embodiments of this disclosure. From Figure 9 It can be seen that the first model consists of three parts: a source adaptive enhancement module, a lightweight encoding and fusion module, and a three-stream differential collaborative decoder, which ultimately outputs the fused salient target image. .

[0093] Source adaptive enhancement module: Includes context-aware saliency reweighting unit and inverse gated structure projection unit, respectively, for the original color visible light image. and original thermal infrared images Preprocessing is performed to output an enhanced color visible light image. and enhanced thermal infrared images Improve the quality of modal input from the source.

[0094] Lightweight coding and fusion module: Contains two structurally identical MobileViT-XS backbone networks, respectively for... and Five-layer feature extraction is performed, and five layers of visible light features are output from... arrive and five layers of thermal infrared features from arrive It contains five lightweight fusion units for each layer. and Channel concatenation and 3×3 convolution are performed to output five layers of fused features. arrive .

[0095] The three-stream differential co-decoder contains three parallel and independent decoding paths: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. Visible light differential decoding path: based on arrive and arrive Layer-by-layer decoding outputs visible light differential saliency prediction feature maps. ; Thermal infrared differential decoding path: based on arrive and arrive Layer-by-layer decoding outputs thermal infrared differential saliency prediction feature maps. ; Collaborative fusion decoding path: based on arrive same level / The system decodes the collaborative features layer by layer, integrates a context-aware decoupling and aggregation module and a dynamic spatial aggregation module, and outputs a collaborative saliency prediction feature map. Finally, the outputs of the three decoders are... , , Add them together to generate the final saliency prediction image, denoted as . .

[0096] To further illustrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method disclosed herein.

[0097] In this embodiment, the method of the present invention is used to test the VT5000 dataset. VT5000 contains 5000 pairs of visible light image pairs with various complex scenes (such as low illumination, thermal crossover, cluttered backgrounds, etc.), wherein the training set includes 2500 pairs of color visible light images and infrared images, and the test set includes another 2500 pairs of color visible light images and infrared images.

[0098] In this embodiment, six commonly used objective evaluation metrics were selected to evaluate the performance of the model, including structure-measure (S-measure), enhanced-alignment measure (E-measure), F-measure, mean absolute error (MAE), number of parameters (Params), and floating-point operations (FLOPs).

[0099] Table 1 below shows the correlation between salient target images and label images obtained using the method of this disclosure on the VT5000 dataset. Table 1: S-measure, E-measure, F-measure, MeanAbsoluteError (MAE), Params, and FLOPs between salient target images and label images obtained using the method of this disclosure on the VT5000 dataset: Table 1

[0100] The results shown in Table 1 demonstrate that the method of this invention achieves high S-measure (0.905), E-measure (0.943), and F-measure (0.864) and low MAE (0.030) on the existing color visible light and infrared image salient target detection dataset (VT5000), while also exhibiting extremely low parameter count (5.26M) and computational cost (4.28 GFLOPs). This indicates that the salient target images and labeled images obtained by the method of this invention are quite close, proving that the method of this invention can effectively achieve high-precision salient target detection in color visible light and infrared images with low parameter count and computational cost.

[0101] This disclosure also provides a salient target detection device. Figure 10 This is a structural block diagram of a salient target detection device provided in an embodiment of the present disclosure, such as... Figure 10 As shown, the salient target detection device 1000 includes: Acquisition unit 1001 is used to acquire a first visible light image and a first thermal infrared image of the object to be detected; Input unit 1002 is used to input a first visible light image and a first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected. The first model is an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the trained object. The first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder. The source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain enhanced third visible light images and third thermal infrared images. The encoding and fusion module is used to extract multi-scale features from the third visible light image and the third thermal infrared image respectively and perform cross-modal feature extraction. The fusion of features yields fused features; the three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second saliency target image; the three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path; the visible light differential decoding path decodes based on the third visible light image and the fused features to obtain visible light differential saliency prediction features; the thermal infrared differential decoding path decodes based on the third thermal infrared image features and the fused features to obtain thermal infrared differential saliency prediction features; the co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fused features to obtain the second saliency target image.

[0102] In one exemplary embodiment, the input unit 1002 is specifically used for: the source adaptive enhancement module includes: a context-aware saliency reweighting unit and an inverse gating structure unit; the context-aware saliency reweighting unit is used to receive a second visible light image and a second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; perform overlapping weighted enhancement on the second visible light image using the guiding features and cooperative gating signals to obtain a third visible light image; the inverse gating structure unit is used to receive and extract gradient features from the second visible light image and the second thermal infrared image respectively; generate inverse gating weights and structural texture features based on the visible light gradient features and the thermal infrared gradient features; perform weighted filtering on the structural texture features based on the inverse gating weights, and project the filtered structural texture features onto the second thermal infrared image for repair enhancement to obtain the third thermal infrared image.

[0103] In one exemplary embodiment, the input unit 1002 is specifically configured as follows: the context-aware saliency reweighting unit includes: a local contrast enhancer, a guiding signal processor, a cooperative gating generator, a first element-wise multiplication layer, and a first adder; wherein, the local contrast enhancer is used to perform contrast processing on the second thermal infrared image to generate a thermal saliency map; the guiding signal processor is used to perform a first convolution and a first normalization process on the thermal saliency map to output guiding features; the cooperative gating generator is used to stitch together the second visible light image and the thermal saliency map, and after performing a second convolution, a second normalization, and a first activation process, generate a cooperative gating signal; the first element-wise multiplication layer is used to multiply the guiding features with the cooperative gating signal to obtain an enhancement term; the first adder is used to scale the enhancement term and superimpose it with the second visible light image to obtain a third visible light image.

[0104] In one exemplary embodiment, the input unit 1002 is specifically used for: the inverse gating structure unit includes: a gradient extraction module, an inverse gating generator, a structure projector, a second element-wise multiplication layer, and a second adder; wherein, the gradient extraction module is used to extract gradient features from the second visible light image and the second thermal infrared image respectively to obtain visible light gradient features and thermal infrared gradient features; the inverse gating generator is used to concatenate the visible light gradient features, thermal infrared gradient features, and the difference between the visible light gradient features and thermal infrared gradient features, and generate inverse gating weights through a third convolution, a third normalization, and a second activation process; the structure projector is used to perform a fourth convolution and a fourth normalization process on the visible light gradient features to output structure texture features; the second element-wise multiplication layer is used to multiply the inverse gating weights and the structure texture features to obtain weighted structure texture features; the second adder is used to scale the weighted structure texture features and superimpose them on the second thermal infrared image to obtain a third thermal infrared image.

[0105] In one exemplary embodiment, the input unit 1002 is specifically used for: the encoding and fusion module includes: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have the same structure and each includes at least five lightweight visual converter layers; for each lightweight visual converter layer, the two MobileViT-XS extract multi-scale features of the third visible light image and the third thermal infrared image respectively, and output the corresponding layer feature map of the image; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; the fusion unit is used to perform channel splicing processing on the third visible light image features and the third thermal infrared image features of the same layer to obtain fused features.

[0106] In one exemplary embodiment, the input unit 1002 is specifically used for: the visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential co-decoder have the same structure, both including: a bilinear interpolation upsampling unit, a channel stitching unit, and a depthwise separable convolutional normalization activation unit; wherein, the bilinear interpolation upsampling unit is used to perform upsampling processing on the third visible light image or the third thermal infrared image respectively, to sequentially obtain the first upsampling feature or the second upsampling feature; the channel stitching unit is used to perform channel stitching on the first upsampling feature and the third visible light image and the fused feature, or the second upsampling feature and the third thermal infrared image and the fused feature, to obtain the first stitched feature or the second stitched feature respectively; the depthwise separable convolutional normalization activation unit is used to perform fifth convolution, fifth normalization and third activation processing on the first stitched feature or the second stitched feature, to correspondingly obtain the visible light differential saliency prediction feature or the thermal infrared differential saliency prediction feature.

[0107] In one exemplary embodiment, the input unit 1002 is specifically used for: the collaborative fusion decoding path in the three-stream differential collaborative decoder includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module; the context-aware decoupling and aggregation module is used to receive and determine a first high-level collaborative feature based on visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fusion features; the modality-aware dynamic aggregation module is used to receive and determine a second saliency target image based on the first high-level collaborative feature, visible light differential saliency prediction features, and thermal infrared differential saliency prediction features.

[0108] In one exemplary embodiment, the input unit 1002 is specifically used for: the context-aware decoupling and aggregation module includes: a boundary attention branch, a region attention branch, a boundary enhancer, a region enhancer, a gated interaction reassembly unit, and a residual connection unit; wherein, the boundary attention branch is used to perform boundary region awareness on the fused features, generate a boundary attention map, and perform boundary weighting on the fused features based on the boundary attention map to obtain a first boundary feature; the region attention branch is used to perform region semantic awareness on the fused features, generate a region attention map, and perform region weighting on the fused features based on the region attention map to obtain a first region feature; the boundary enhancer is used to perform directional enhancement on the first boundary feature to obtain a second boundary feature; the region enhancer is used to perform semantic enhancement on the first region feature to obtain a second region feature; the gated interaction reassembly unit is used to perform gated weighted interaction on the second boundary feature and the second region feature to obtain a reassembled feature; the residual connection unit is used to superimpose the reassembled feature and the fused feature to obtain a first high-level collaborative feature.

[0109] In one exemplary embodiment, the input unit 1002 is specifically used for: the modality-aware dynamic aggregation module includes: an initial fusion unit, a spatial weight generation unit, a detail enhancement branch, a structure enhancement branch, and a weighted summation unit; wherein, the initial fusion unit is used to add the first high-level collaborative feature, the visible light differential saliency prediction feature, and the thermal infrared differential saliency prediction feature element-wise to obtain the second high-level collaborative feature; the spatial weight generation unit is used to perform a sixth convolution, a sixth normalization, and a fourth activation processing on the second high-level collaborative feature to generate a spatial weight map; the detail enhancement branch is used to extract the texture information of the second high-level collaborative processing to obtain texture features; the structure enhancement branch is used to superimpose the first high-level collaborative feature and the second high-level collaborative feature to obtain structural features; the weighted summation unit is used to use the spatial weight map to perform weighted summation on the texture features and the structural features respectively to obtain the second saliency target image.

[0110] This disclosure also provides a system for salient target detection, the system comprising: The source adaptive enhancement module is used to receive the first visible light image and the first thermal infrared image of the object to be detected, and to perform complementary enhancement on the first visible light image and the first thermal infrared image to obtain the enhanced fourth visible light image and the fourth thermal infrared image. The encoding and fusion module is used to extract multi-scale features from the fourth visible light image and the fourth thermal infrared image respectively, and perform cross-modal feature fusion to obtain the fused features of the object to be detected; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; A three-stream differential co-decoder is used to decode the fused features of a fourth visible light image, a fourth thermal infrared image, and the object to be detected through three parallel decoding paths to obtain a first salient target image. The three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The visible light differential decoding path decodes based on the fused features of the fourth visible light image and the object to be detected to obtain the visible light differential saliency prediction features of the object to be detected. The thermal infrared differential decoding path decodes based on the features of the fourth thermal infrared image and the fused features of the object to be detected to obtain the thermal infrared differential saliency prediction features of the object to be detected. The co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fused features of the object to be detected to obtain the first salient target image.

[0111] Figure 11This is a hardware block diagram of an electronic device provided according to an embodiment of the present disclosure. The electronic device 1100 according to an embodiment of the present disclosure includes at least a processor and a memory for storing computer-readable instructions. When the computer-readable instructions are loaded and executed by the processor, the processor performs the salient target detection method described in any of the preceding embodiments of the present disclosure.

[0112] Figure 11 The illustrated electronic device 1100 specifically includes a central processing unit (CPU) 1101, a graphics processing unit (GPU) 1102, and a memory 1103. These units are interconnected via a bus 1104. The CPU 1101 and / or GPU 1102 can function as the aforementioned processor, and the memory 1103 can function as the aforementioned memory storing computer-readable instructions. Furthermore, the electronic device 1100 may also include a communication unit 1105, a storage unit 1106, an output unit 1107, an electronic device input unit 1108, and an external device 1109, all of which are also connected to the bus 1104.

[0113] In summary, this disclosure provides a method, apparatus, system, and electronic device for salient target detection. This disclosure involves acquiring a first visible light image and a first thermal infrared image of an object to be detected; inputting the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected; the first model is an image fusion neural network model determined based on a second visible light image and a second thermal infrared image of the trained object; the first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential cooperative decoder; the source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain enhanced third visible light images and third thermal infrared images; the encoding and fusion module is used to extract multi-scale values ​​from the third visible light image and the third thermal infrared image respectively. Features are fused across modalities to obtain fused features; a three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain the second saliency target image; the three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path; the visible light differential decoding path decodes based on the third visible light image and fused features to obtain visible light differential saliency prediction features; the thermal infrared differential decoding path decodes based on the third thermal infrared image features and fused features to obtain thermal infrared differential saliency prediction features; the co-fusion decoding path decodes based on the visible light differential saliency prediction features, thermal infrared differential saliency prediction features, and fused features to obtain the second saliency target image. In this way, compared with existing single-modality detection methods and simple fusion detection methods, this disclosure can specifically address the following issues: For the problems of weak anti-interference capability of single visible light detection and blurred boundaries of single thermal infrared detection, this disclosure enhances the effective complementary enhancement of both through a source adaptive enhancement module. Specifically, it utilizes the thermal radiation characteristics of the thermal infrared image to perform overlapping weighted enhancement on the visible light image, suppressing background clutter and illumination interference. Simultaneously, it uses the high-frequency structural features of the visible light image to repair the blurred boundaries of the thermal infrared image and alleviate thermal cross-interference, thereby improving the input quality of both modalities from the source. Furthermore, for the problem of simple fusion detection ignoring modal differences and resulting in coarse fusion, this disclosure can extract multi-scale features of the two enhanced modalities separately through encoding and fusion modules, achieving accurate cross-modal feature integration. Moreover, through the three parallel paths of the three-stream differential collaborative decoder, it refines the decoding of the two modal features and the fused features, and then achieves dynamic synergy of the predicted features of the two modalities through a collaborative fusion path, further improving detection accuracy. In summary, the technical solution provided in this disclosure can address the technical problems of existing methods, such as low detection accuracy, weak anti-interference ability, and poor fusion effect, by addressing the three core aspects of input enhancement, feature fusion, and decoding collaboration in a progressive manner. This improves the accuracy of salient target detection and can be adapted to various application scenarios.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0115] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0116] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0117] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0118] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0119] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0120] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0121] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for salient target detection, characterized in that, The method includes: Acquire the first visible light image and the first thermal infrared image of the object to be detected; The first visible light image and the first thermal infrared image are input into a pre-trained first model to obtain a first salient target image of the object to be detected. The first model is an image fusion neural network model determined based on the second visible light image and the second thermal infrared image of the trained object. The first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder. The source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain enhanced third visible light images and third thermal infrared images. The encoding and fusion module is used to extract multi-scale features from the third visible light image and the third thermal infrared image respectively and perform cross-modal feature fusion to obtain fused features. The three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain a second saliency target image. The three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The visible light differential decoding path decodes based on the third visible light image and the fused features to obtain visible light differential saliency prediction features. The thermal infrared differential decoding path decodes based on the third thermal infrared image features and the fused features to obtain thermal infrared differential saliency prediction features. The co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fused features to obtain the second saliency target image.

2. The method according to claim 1, characterized in that, The source adaptive enhancement module includes: a context-aware saliency reweighting unit and an inverse gating structure unit; The context-aware saliency reweighting unit is used to receive the second visible light image and the second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; and perform overlapping weighted enhancement on the second visible light image using the guiding features and the cooperative gating signals to obtain the third visible light image. The reverse gating structure unit is used to receive and extract gradient features of the second visible light image and the second thermal infrared image respectively; based on the visible light gradient features and thermal infrared gradient features, reverse gating weights and structural texture features are generated; the structural texture features are weighted and filtered based on the reverse gating weights, and the filtered structural texture features are projected onto the second thermal infrared image for repair and enhancement to obtain the third thermal infrared image.

3. The method according to claim 2, characterized in that, The context-aware saliency reweighting unit includes: a local contrast enhancer, a guiding signal processor, a cooperative gating generator, a first element-wise multiplication layer, and a first adder; The local contrast enhancer is used to perform contrast processing on the second thermal infrared image to generate the thermal saliency map; the guiding signal processor is used to perform a first convolution and a first normalization process on the thermal saliency map to output the guiding feature; the cooperative gating generator is used to stitch the second visible light image and the thermal saliency map together, and then perform a second convolution, a second normalization, and a first activation process to generate the cooperative gating signal; the first element-wise multiplication layer is used to multiply the guiding feature with the cooperative gating signal to obtain an enhancement term; the first adder is used to scale the enhancement term and then superimpose it with the second visible light image to obtain the third visible light image.

4. The method according to claim 2, characterized in that, The reverse gated structure unit includes: a gradient extraction module, a reverse gated generator, a structure projector, a second element multiplication layer, and a second adder; The gradient extraction module is used to extract gradient features from the second visible light image and the second thermal infrared image respectively, to obtain the visible light gradient features and the thermal infrared gradient features; the inverse gating generator is used to concatenate the visible light gradient features, the thermal infrared gradient features, and the difference between the visible light gradient features and the thermal infrared gradient features, and generate the inverse gating weights through a third convolution, a third normalization, and a second activation process; the structure projector is used to perform a fourth convolution and a fourth normalization process on the visible light gradient features to output the structure texture features; the second element-wise multiplication layer is used to multiply the inverse gating weights and the structure texture features to obtain weighted structure texture features; the second adder is used to scale the weighted structure texture features and superimpose them with the second thermal infrared image to obtain the third thermal infrared image.

5. The method according to claim 1, characterized in that, The encoding and fusion module includes: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have the same structure and each includes at least five lightweight visual converter layers; For each lightweight visual converter layer, the two MobileViT-XS extract the multi-scale features of the third visible light image and the third thermal infrared image respectively, and output the corresponding layer feature map of the image; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; The fusion unit is used to perform channel stitching processing on the third visible light image feature and the third thermal infrared image feature at the same level to obtain the fused feature.

6. The method according to claim 1, characterized in that, The visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential collaborative decoder have the same structure, both including: a bilinear interpolation upsampling unit, a channel splicing unit, and a depthwise separable convolutional normalization activation unit; The bilinear interpolation upsampling unit is used to upsample the third visible light image or the third thermal infrared image respectively to obtain a first upsampled feature or a second upsampled feature in sequence; the channel stitching unit is used to stitch the first upsampled feature and the third visible light image and the fused feature, or the second upsampled feature and the third thermal infrared image and the fused feature, to obtain a first stitched feature or a second stitched feature respectively; the depth-separable convolutional normalization activation unit is used to perform a fifth convolution, a fifth normalization and a third activation on the first stitched feature or the second stitched feature to obtain the visible light differential saliency prediction feature or the thermal infrared differential saliency prediction feature.

7. The method according to claim 1, characterized in that, The collaborative fusion decoding path in the three-stream differential collaborative decoder includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module; The context-aware decoupling and aggregation module is used to receive and determine the first high-level collaborative feature based on the visible light differential saliency prediction feature, the thermal infrared differential saliency prediction feature, and the fused feature; The modality-aware dynamic aggregation module is used to receive and determine the second salient target image based on the first high-level collaborative features, the visible light differential saliency prediction features, and the thermal infrared differential saliency prediction features.

8. The method according to claim 7, characterized in that, The context-aware decoupling and aggregation module includes: a boundary attention branch, a region attention branch, a boundary enhancer, a region enhancer, a gated interaction reassembly unit, and a residual connection unit; The boundary attention branch is used to perceive the boundary regions of the input features, generate a boundary attention map, and perform boundary weighting on the input features based on the boundary attention map to obtain a first boundary feature; the region attention branch is used to perceive the region semantics of the input features, generate a region attention map, and perform region weighting on the input features based on the region attention map to obtain a first region feature; the boundary enhancer is used to enhance the directionality of the first boundary feature to obtain a second boundary feature; the region enhancer is used to enhance the semantics of the first region feature to obtain a second region feature; the gated interaction recombination unit is used to perform gated weighted interaction on the second boundary feature and the second region feature to obtain a recombined feature; the residual connection unit is used to superimpose the recombined feature and the input feature to obtain a first high-level collaborative feature; the input feature is the fused feature or the spliced ​​feature; the spliced ​​feature is determined based on the recombined feature.

9. The method according to claim 7, characterized in that, The modality-aware dynamic aggregation module includes: an initial fusion unit, a spatial weight generation unit, a detail enhancement branch, a structure enhancement branch, and a weighted summation unit; The initial fusion unit is used to add the first high-level collaborative feature, the visible light differential saliency prediction feature, and the thermal infrared differential saliency prediction feature element-wise to obtain the second high-level collaborative feature; the spatial weight generation unit is used to perform a sixth convolution, a sixth normalization, and a fourth activation on the second high-level collaborative feature to generate a spatial weight map; the detail enhancement branch is used to extract the texture information of the second high-level collaborative processing to obtain texture features; the structure enhancement branch is used to superimpose the first high-level collaborative feature and the second high-level collaborative feature to obtain structural features; and the weighted summation unit is used to perform weighted summation on the texture features and the structural features using the spatial weight map to obtain the second saliency target image.

10. A device for detecting salient targets, characterized in that, The device includes: The acquisition unit is used to acquire a first visible light image and a first thermal infrared image of the object to be detected. The input unit is used to input the first visible light image and the first thermal infrared image into a pre-trained first model to obtain a first salient target image of the object to be detected. The first model is an image fusion neural network model determined based on the second visible light image and the second thermal infrared image of the trained object. The first model includes: a source adaptive enhancement module, an encoding and fusion module, and a three-stream differential collaborative decoder. The source adaptive enhancement module is used to perform complementary enhancement based on the second visible light image and the second thermal infrared image to obtain enhanced third visible light images and third thermal infrared images. The encoding and fusion module is used to extract multi-scale features from the third visible light image and the third thermal infrared image respectively and perform cross-modal feature fusion to obtain... The three-stream differential co-decoder is used to decode through three parallel decoding paths to obtain a second saliency target image. The three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The visible light differential decoding path decodes based on the third visible light image and the fusion features to obtain visible light differential saliency prediction features. The thermal infrared differential decoding path decodes based on the third thermal infrared image features and the fusion features to obtain thermal infrared differential saliency prediction features. The co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fusion features to obtain the second saliency target image.

11. The apparatus according to claim 10, characterized in that, The source adaptive enhancement module includes: a context-aware saliency reweighting unit and an inverse gating structure unit; The context-aware saliency reweighting unit is used to receive the second visible light image and the second thermal infrared image; generate a thermal saliency map based on the second thermal infrared image; generate guiding features and cooperative gating signals based on the thermal saliency map; and perform overlapping weighted enhancement on the second visible light image using the guiding features and the cooperative gating signals to obtain the third visible light image. The reverse gating structure unit is used to receive and extract gradient features of the second visible light image and the second thermal infrared image respectively; based on the visible light gradient features and thermal infrared gradient features, reverse gating weights and structural texture features are generated; the structural texture features are weighted and filtered based on the reverse gating weights, and the filtered structural texture features are projected onto the second thermal infrared image for repair and enhancement to obtain the third thermal infrared image.

12. The apparatus according to claim 10, characterized in that, The encoding and fusion module includes: two lightweight visual converter networks MobileViT-XS and at least five fusion units; the two MobileViT-XS have the same structure and each includes at least five lightweight visual converter layers; For each lightweight visual converter layer, the two MobileViT-XS extract the multi-scale features of the third visible light image and the third thermal infrared image respectively, and output the corresponding layer feature map of the image; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; The fusion unit is used to perform channel stitching processing on the third visible light image feature and the third thermal infrared image feature at the same level to obtain the fused feature.

13. The apparatus according to claim 10, characterized in that, The visible light differential decoding path and the thermal infrared differential decoding path in the three-stream differential collaborative decoder have the same structure, both including: a bilinear interpolation upsampling unit, a channel splicing unit, and a depthwise separable convolutional normalization activation unit; The bilinear interpolation upsampling unit is used to upsample the third visible light image or the third thermal infrared image respectively to obtain a first upsampled feature or a second upsampled feature in sequence; the channel stitching unit is used to stitch the first upsampled feature and the third visible light image and the fused feature, or the second upsampled feature and the third thermal infrared image and the fused feature, to obtain a first stitched feature or a second stitched feature respectively; the depth-separable convolutional normalization activation unit is used to perform a fifth convolution, a fifth normalization and a third activation on the first stitched feature or the second stitched feature to obtain the visible light differential saliency prediction feature or the thermal infrared differential saliency prediction feature.

14. The apparatus according to claim 10, characterized in that, The collaborative fusion decoding path in the three-stream differential collaborative decoder includes: a context-aware decoupling and aggregation module, and a modality-aware dynamic aggregation module; The context-aware decoupling and aggregation module is used to receive and determine the first high-level collaborative feature based on the visible light differential saliency prediction feature, the thermal infrared differential saliency prediction feature, and the fused feature; The modality-aware dynamic aggregation module is used to receive and determine the second salient target image based on the first high-level collaborative features, the visible light differential saliency prediction features, and the thermal infrared differential saliency prediction features.

15. A system for detecting salient targets, characterized in that, The system includes: The source adaptive enhancement module is used to receive a first visible light image and a first thermal infrared image of the object to be detected, and to perform complementary enhancement on the first visible light image and the first thermal infrared image to obtain an enhanced fourth visible light image and a fourth thermal infrared image. The encoding and fusion module is used to extract multi-scale features from the fourth visible light image and the fourth thermal infrared image respectively, and perform cross-modal feature fusion to obtain the fused features of the object to be detected; the multi-scale features include: image texture features, image contour features, local semantic features, global semantic features and deep semantic features; A three-stream differential co-decoder is used to decode the fusion features of the fourth visible light image, the fourth thermal infrared image, and the object to be detected through three parallel decoding paths to obtain a first salient target image. The three parallel decoding paths include: a visible light differential decoding path, a thermal infrared differential decoding path, and a co-fusion decoding path. The visible light differential decoding path decodes based on the fusion features of the fourth visible light image and the object to be detected to obtain visible light differential saliency prediction features of the object to be detected. The thermal infrared differential decoding path decodes based on the features of the fourth thermal infrared image and the fusion features of the object to be detected to obtain thermal infrared differential saliency prediction features of the object to be detected. The co-fusion decoding path decodes based on the visible light differential saliency prediction features, the thermal infrared differential saliency prediction features, and the fusion features of the object to be detected to obtain the first salient target image.

16. An electronic device, characterized in that, include: Memory, used to store computer-readable instructions; as well as A processor for executing the computer-readable instructions, causing the electronic device to perform the method as described in any one of claims 1-9.

Images