Color visible and infrared image salient object detection method

Abstract

This invention discloses a method for salient target detection using color visible light and infrared images. It constructs a training set containing several pairs of color visible light images and their corresponding infrared images, and builds a neural network. The neural network mainly consists of a feature extraction module, an asymmetric feature complementation module, a feature selection and fusion module, and a semantic enhancement decoder. The asymmetric feature complementation module comprises a color visible light feature complementation module and an infrared feature complementation module. The several pairs of color visible light images and their corresponding infrared images from the training set are input into the neural network for multiple rounds of training. After training, a neural network training model is obtained. The neural network training model is used to predict the salient target images of test image pairs. The advantage is that it can effectively improve the accuracy of salient target detection.

Description

Technical Field

[0001] This invention relates to a salient object detection technology, and more particularly to a method for salient target detection in color visible light and infrared images. Background Technology

[0002] Salient Object Detection (SOD) aims to capture and segment salient objects in images or videos. As an important preprocessing step, SOD has been widely applied in computer vision and image processing tasks such as image segmentation, object tracking, image retrieval, and image quality assessment. In recent years, convolutional / deep neural networks have pushed the performance of SOD to new heights due to their powerful learning capabilities and excellent performance in feature extraction. However, when faced with challenging scenes (such as multiple objects, cluttered backgrounds, unfavorable lighting conditions, or transparent objects), the performance of SOD using only color visible light images degrades. Therefore, incorporating other modalities and combining them with color visible light images is a common solution in SOD tasks.

[0003] Saliency target detection using color visible light and depth images with auxiliary depth images, and color visible light and infrared images with infrared images, have been developed, utilizing a wide range of depth sensors and infrared cameras as supplementary modal information. While depth information contains rich spatial structure and 3D layout information, it is unreliable for providing useful information for salient target detection in some extreme environments (e.g., poor lighting and cluttered scenes). Since infrared images can reflect the thermal radiation of object surfaces, they naturally complement color visible light images in these extreme environments, and research on salient target detection using color visible light and infrared images has received increasing attention in recent years.

[0004] Currently, the following problems still exist in multimodal saliency target detection:

[0005] First, how can a reasonable multimodal feature interaction strategy be adopted to handle the differences between the two modalities? Ideally, both color visible light images and infrared images can provide rich geometric information and object boundaries. Furthermore, color visible light images can provide sufficient color information under good lighting conditions, while infrared images can provide temperature information about the object's surface. This leads to differences between the two modalities, and in the process of identifying salient targets, people tend to determine the target's location using color visible light images rather than infrared images. At the same time, infrared images are more likely to provide interfering information, thus affecting target identification. Therefore, to reduce the introduction of interfering information and make the best use of the information from both modalities, an asymmetric feature interaction strategy is more reasonable.

[0006] Second, how can we balance the internal differences between different modalities and fuse multimodal features? After feature interaction, since these single-modal features only carry partial discriminative information from another modality in addition to their own modal information, it is necessary to study an effective fusion strategy to fully utilize the complementary information across modalities. Most existing methods use simple means such as linear operations, convolution operations, attention methods, or other combinations of methods, which fail to fully utilize the complementary information between the two modalities. Therefore, selecting and fusing valuable features within different modalities is essential for improving the accuracy of salient target detection in color visible light and infrared images. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method for detecting salient targets in color visible light and infrared images, which can effectively improve the accuracy of salient target detection.

[0008] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a method for detecting salient targets in color visible light and infrared images. This method first constructs a training set containing several pairs of color visible light images and their corresponding infrared images, and builds a neural network. Secondly, the several pairs of color visible light images and their corresponding infrared images in the training set are input into the neural network for multiple rounds of network training. After the network training is completed, a neural network training model is obtained. Thirdly, the neural network training model is used to predict test image pairs, predicting the salient target image of the test image pair. The neural network is characterized by mainly consisting of a feature extraction module, an asymmetric feature complementation module, a feature selection and fusion module, and a semantic enhancement decoder, wherein:

[0009] The feature extraction module includes two ResNet50 backbone networks; the input of the first layer of the first ResNet50 backbone network receives a color visible light image of size H×W×3, the output of the first layer of the first ResNet50 backbone network is denoted as FR1, the input of the second layer of the first ResNet50 backbone network receives FR1, the output of the second layer of the first ResNet50 backbone network is denoted as FR2, the input of the third layer of the first ResNet50 backbone network receives FR2, the output of the third layer of the first ResNet50 backbone network is denoted as FR3, the input of the fourth layer of the first ResNet50 backbone network receives FR3, the output of the fourth layer of the first ResNet50 backbone network is denoted as FR4, the input of the fifth layer of the first ResNet50 backbone network receives FR4, and the output of the fifth layer of the first ResNet50 backbone network is denoted as FR5; The first layer of two ResNet50 backbone networks receives an infrared image of size H×W×3 as its input. The feature map output from the first layer of the second ResNet50 backbone network is denoted as FT1. The second layer of the second ResNet50 backbone network receives FT1 as its input, and the feature map output from the second layer of the second ResNet50 backbone network is denoted as FT2. The third layer of the second ResNet50 backbone network receives FT2 as its input. The feature map output from the third layer of the network is denoted as FT3. The input of the fourth layer of the second ResNet50 backbone network receives FT3, and the output of the fourth layer of the second ResNet50 backbone network is denoted as FT4. The input of the fifth layer of the second ResNet50 backbone network receives FT4, and the output of the fifth layer of the second ResNet50 backbone network is denoted as FT5. The ResNet50 backbone network has five layers. The sizes of FR1 and FT1 are... The sizes of FR2 and FT2 are The sizes of FR3 and FT3 are The sizes of FR4 and FT4 are The sizes of FR5 and FT5 are

[0010] The asymmetric feature complementation module is composed of a color visible light feature complementation module and an infrared feature complementation module;

[0011] The aforementioned color visible light feature complementation module comprises five identical color visible light feature complementation blocks; the first input terminal of the first color visible light feature complementation block receives FR1, the second input terminal receives FT1, and the feature map output from the output terminal of the first color visible light feature complementation block is denoted as... The first input of the second color visible light feature complementary block receives FR2, the second input receives FT2, and the feature map output from the output of the second color visible light feature complementary block is denoted as follows. The first input of the third color visible light feature complement block receives FR3, the second input receives FT3, and the feature map output from the output of the third color visible light feature complement block is denoted as follows. The first input of the fourth color visible light feature complementary block receives FR4, the second input receives FT4, and the feature map output from the output of the fourth color visible light feature complementary block is denoted as follows. The first input of the fifth color visible light feature complementary block receives FR5, the second input receives FT5, and the feature map output from the output of the fifth color visible light feature complementary block is denoted as follows. in, The size is The size is The size is The size is The size is

[0012] The infrared feature complementation module comprises five identical infrared feature complementation blocks; the first input terminal of the first infrared feature complementation block receives FT1, the second input terminal receives FR1, and the feature map output from the output terminal of the first infrared feature complementation block is denoted as... The first input terminal of the second infrared feature complement block receives FT2, the second input terminal receives FR2, and the feature map output by the output terminal of the second infrared feature complement block is denoted as... The first input terminal of the third infrared feature complement block receives FT3, the second input terminal receives FR3, and the feature map output by the output terminal of the third infrared feature complement block is denoted as... The first input terminal of the fourth infrared feature complement block receives FT4, the second input terminal receives FR4, and the feature map output from the output terminal of the fourth infrared feature complement block is denoted as follows. The first input terminal of the fifth infrared feature complementary block receives FT5, the second input terminal receives FR5, and the feature map output from the output terminal of the fifth infrared feature complementary block is denoted as... in, The size is The size is The size is The size is The size is

[0013] The feature selection and fusion module includes five structurally identical feature selection and fusion blocks; the first input terminal of the first feature selection and fusion block receives... The second input terminal receives The feature map output from the first feature selection and fusion block is denoted as follows: The first input terminal of the second feature selection and fusion block receives... The second input terminal receives The feature map output from the output of the second feature selection and fusion block is denoted as follows: The first input terminal of the third feature selection and fusion block receives... The second input terminal receives The feature map output from the third feature selection and fusion block is denoted as follows: The first input terminal of the fourth feature selection and fusion block receives The second input terminal receives The feature map output from the output of the fourth feature selection and fusion block is denoted as follows: The first input terminal of the fifth feature selection and fusion block receives... The second input terminal receives The feature map output by the 5th feature selection and fusion block is denoted as follows: in, The size is The size is The size is The size is The size is

[0014] The semantic enhancement decoder includes three structurally identical semantic enhancement decoding blocks; the first input of the first semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output by the output of the first semantic enhancement decoding block is denoted as S1, and S1 is used as the first coarse saliency target image; the first input of the second semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the output of the second semantic enhancement decoding block is denoted as S2, and S2 is used as the second coarse saliency target image; the first input of the third semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the third semantic enhancement decoding block is denoted as S3, and S3 is used as the third coarse saliency target image; where the size of S1 is... The size of S2 is The size of S3 is

[0015] The first coarse saliency target image S1, the second coarse saliency target image S2, and the third coarse saliency target image S3 are summed element-wise and then upsampled. The resulting feature map is denoted as S. f and S f As the final salient target image; where the upsampling operation mode is bilinear, S f Its size is H×W×1.

[0016] The training set is constructed as follows: select at least 200 pairs of original color visible light images and their corresponding original infrared images; then perform downsampling operation on each original color visible light image and its corresponding original infrared image to downsample the image size to H×W; and then construct the training set from all the color visible light images and their corresponding infrared images of size H×W.

[0017] The process of obtaining the neural network training model is as follows: Each pair of color visible light images and their corresponding infrared images from the training set are input into the neural network for training. Before the end of each training round, the loss function L is calculated to optimize the neural network. After a total of 150 training rounds, the neural network training model is obtained. Specifically, in the first 40 rounds of training… In the last 110 rounds of network training S f Let G represent the final salient target image output by the neural network, and S represent the label image. j Let L represent the j-th coarse saliency target image obtained in the neural network. bce () represents the binary cross-entropy loss, L iou () indicates the intersection and union ratio loss.

[0018] The process of using a neural network training model to predict the salient target image of a test image pair is as follows: arbitrarily select a pair of original color visible light images and their corresponding original infrared images; then downsample the pair of original color visible light images and original infrared images to reduce the image size to H×W, and use them as the test image pair; then input the test image pair into the neural network training model to predict the salient target image of the test image pair.

[0019] The color visible light feature complement block mainly consists of the first to fifth convolutional layers, the first to fifth batch normalization layers, and the first to fifth ReLU activation layers. For the i-th color visible light feature complement block, the input of the first convolutional layer serves as the first input of the i-th color visible light feature complement block, receiving FR. i The input of the first Batch Normalization layer receives the feature map output from the first convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first Batch Normalization layer. The feature map output from the first ReLU activation layer is denoted as... The input of the second convolutional layer serves as the second input of the i-th color visible light feature complementary block, receiving the FT. i The input of the second Batch Normalization layer receives the feature map output from the second convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second Batch Normalization layer. The feature map output from the second ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform channel connection operations and denote the resulting feature map as... The input of the third convolutional layer receives The input of the third Batch Normalization layer receives the feature map output from the third convolutional layer, and the input of the third ReLU activation layer receives the feature map output from the third Batch Normalization layer. The feature map output from the third ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the fourth convolutional layer receives The input of the fourth Batch Normalization layer receives the feature map output from the fourth convolutional layer, and the input of the fourth ReLU activation layer receives the feature map output from the fourth Batch Normalization layer. The feature map output from the fourth ReLU activation layer is denoted as... The input of the 5th convolutional layer receives FR i The input of the 5th Batch Normalization layer receives the feature map output from the 5th convolutional layer, and the input of the 5th ReLU activation layer receives the feature map output from the 5th Batch Normalization layer. The feature map output from the 5th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th color visible light feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The first to fifth convolutional layers are all 2D convolutional layers. In the first color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the second color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the third color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 4th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 5th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the autofill is 1.

[0020] The infrared feature complement block mainly consists of the 6th to 9th convolutional layers, the 6th to 9th batch normalization layers, the 6th to 9th ReLU activation layers, and the 1st spatial attention layer. For the i-th infrared feature complement block, the input of the 6th convolutional layer serves as the first input of the i-th infrared feature complement block to receive the Fourier Transform (FT). i The input of the 6th Batch Normalization layer receives the feature map output from the 6th convolutional layer, and the input of the 6th ReLU activation layer receives the feature map output from the 6th Batch Normalization layer. The feature map output from the 6th ReLU activation layer is denoted as... The input of the 7th convolutional layer serves as the second input of the i-th infrared feature complementary block to receive FR. i The input of the 7th Batch Normalization layer receives the feature map output from the 7th convolutional layer, and the input of the 7th ReLU activation layer receives the feature map output from the 7th Batch Normalization layer. The feature map output from the 7th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform channel connection operations and denote the resulting feature map as... The input of the 8th convolutional layer receives The input of the 8th Batch Normalization layer receives the feature map output from the 8th convolutional layer, and the input of the 8th ReLU activation layer receives the feature map output from the 8th Batch Normalization layer. The feature map output from the 8th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the 9th convolutional layer receives The input of the 9th Batch Normalization layer receives the feature map output from the 9th convolutional layer, and the input of the 9th ReLU activation layer receives the feature map output from the 9th Batch Normalization layer. The feature map output from the 9th ReLU activation layer is denoted as... The input of the first spatial attention layer receives FR i The feature map output by the first spatial attention layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th infrared feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The 6th to 9th convolutional layers are all 2D convolutional layers. In the first infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the second infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the third infrared feature complement block: the input size of the sixth and seventh convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 4th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 5th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the autofill is 1, and the dilation rate is 1.

[0021] The feature selection and fusion block mainly consists of the first average pooling layer, the second average pooling layer, the first max pooling layer, the second max pooling layer, the first fully connected layer, the second fully connected layer, the first sigmoid activation layer, the second sigmoid activation layer, the 10th to 19th convolutional layers, the 10th to 17th batch normalization layers, the 10th to 12th ReLU activation layers, and the first to 6th PReLU activation layers. For the i-th feature selection and fusion block, the inputs of the first average pooling layer and the first max pooling layer are both received as the first input of the i-th feature selection and fusion block. The feature map output from the first average pooling layer is denoted as... The feature map output from the first max pooling layer is denoted as follows: The input terminals of the second average pooling layer and the second max pooling layer are both received as the second input terminal of the i-th feature selection and fusion block. The feature map output from the second max-pooling layer is denoted as... The feature map output from the output of the second average pooling layer is denoted as... right and Perform channel connection operations and denote the resulting feature map as... right and Perform channel connection operations and denote the resulting feature map as... The input of the first fully connected layer receives... The feature map output from the first fully connected layer is denoted as... The input of the second fully connected layer receives... The feature map output from the output of the second fully connected layer is denoted as... right and Perform element-wise addition and use the resulting feature map as input to the first Sigmoid activation layer. Then, perform channel segmentation on the feature map output from the first Sigmoid activation layer and denote the resulting feature maps as follows: and right and Perform channel-by-channel multiplication and denote the resulting feature map as... right and Perform channel-by-channel multiplication and denote the resulting feature map as... The input of the 10th convolutional layer receives The input of the 10th Batch Normalization layer receives the feature map output from the 10th convolutional layer, and the input of the 1st PReLU activation layer receives the feature map output from the 10th Batch Normalization layer. The feature map output from the 1st PReLU activation layer is denoted as... The input of the 11th convolutional layer receives The input of the 11th Batch Normalization layer receives the feature map output from the 11th convolutional layer, and the input of the 2nd PReLU activation layer receives the feature map output from the 11th Batch Normalization layer. The feature map output from the 2nd PReLU activation layer is denoted as... The input of the 12th convolutional layer receives The input of the 12th Batch Normalization layer receives the feature map output from the 12th convolutional layer, and the input of the 3rd PReLU activation layer receives the feature map output from the 12th Batch Normalization layer. The feature map output from the 3rd PReLU activation layer is denoted as... The input of the 13th convolutional layer receives The input of the 13th Batch Normalization layer receives the feature map output from the 13th convolutional layer, and the input of the 4th PReLU activation layer receives the feature map output from the 13th Batch Normalization layer. The feature map output from the 4th PReLU activation layer is denoted as... The input of the 14th convolutional layer receives The input of the 14th Batch Normalization layer receives the feature map output from the 14th convolutional layer, and the input of the 5th PReLU activation layer receives the feature map output from the 14th Batch Normalization layer. The feature map output from the 5th PReLU activation layer is denoted as... The input of the 15th convolutional layer receives The input of the 15th Batch Normalization layer receives the feature map output from the 15th convolutional layer, and the input of the 6th PReLU activation layer receives the feature map output from the 15th Batch Normalization layer. The feature map output from the 6th PReLU activation layer is denoted as... right and Perform channel connection operations and denote the resulting feature map as... right and Perform channel connection operations and denote the resulting feature map as... The input of the 16th convolutional layer receives The input of the 16th Batch Normalization layer receives the feature map output from the 16th convolutional layer, and the input of the 10th ReLU activation layer receives the feature map output from the 16th Batch Normalization layer. The feature map output from the 10th ReLU activation layer is denoted as... The input of the 17th convolutional layer receives The input of the 17th Batch Normalization layer receives the feature map output from the 17th convolutional layer, and the input of the 11th ReLU activation layer receives the feature map output from the 17th Batch Normalization layer. The feature map output from the 11th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right Perform channel connection operations and denote the resulting feature map as... The input of the 18th convolutional layer receives The input of the 12th ReLU activation layer receives the feature map output from the 18th convolutional layer, and the input of the 19th convolutional layer receives the feature map output from the 12th ReLU activation layer. The feature map output from the 19th convolutional layer is denoted as... The input of the second Sigmoid activation layer receives The feature maps obtained by channel segmentation of the feature map output from the second Sigmoid activation layer are denoted as follows: and right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output by the i-th feature selection and fusion block; where 1 ≤ i ≤ 5. This represents the feature map output from the output terminal of the i-th infrared feature complementary block. This represents the feature map output from the output of the i-th color visible light feature complementary block. The 10th to 19th convolutional layers are all 2D convolutional layers. In the first feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the second feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×512, the size of the intermediate hidden layers is 1×1×256, and the output size is 1×1×512. and The size is 1×1×256, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the third feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The kernel size is 1×1.

[0022] The sliding step size is 1, the auto-fill value is 0, and the expansion rate is 1. and The size is In the fourth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the fifth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is

[0023] The semantic enhancement decoding block mainly consists of convolutional layers 20 to 29, batch normalization layers 18 to 23, ReLU activation layers 13 to 19, sigmoid activation layer 3, a Transformer layer, a spatial attention layer 2, a channel attention layer 1, a channel attention layer 2, and upsampling layers 1 to 3. For the j-th semantic enhancement decoding block, the input of the 20th convolutional layer is used as the first input of the j-th semantic enhancement decoding block. The input of the 18th Batch Normalization layer receives the feature map output from the 20th convolutional layer; the input of the 13th ReLU activation layer receives the feature map output from the 18th Batch Normalization layer; the input of the 1st upsampling layer receives the feature map output from the 13th ReLU activation layer; the input of the 21st convolutional layer receives the feature map output from the 1st upsampling layer; the input of the 19th Batch Normalization layer receives the feature map output from the 21st convolutional layer; the input of the 14th ReLU activation layer receives the feature map output from the 19th Batch Normalization layer; the input of the Transformer layer receives the feature map output from the 14th ReLU activation layer; and the feature map output by the Transformer layer is denoted as... The input terminal of the second spatial attention layer receives The input of the second upsampling layer receives the feature map output from the output of the second spatial attention layer. The feature map output from the second upsampling layer is denoted as... The second input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... The input of the 22nd convolutional layer receives The input of the 20th Batch Normalization layer receives the feature map output from the 22nd convolutional layer. The input of the 15th ReLU activation layer receives the feature map output from the 20th Batch Normalization layer. The input of the 3rd upsampling layer receives the feature map output from the 15th ReLU activation layer. The input of the 23rd convolutional layer receives the feature map output from the 3rd upsampling layer. The input of the 21st Batch Normalization layer receives the feature map output from the 23rd convolutional layer. The input of the 16th ReLU activation layer receives the feature map output from the 21st Batch Normalization layer. The feature map output from the 16th ReLU activation layer is denoted as... right and Perform channel stitching and denote the resulting feature map as... The first channel of the attention layer receives input. The input of the 24th convolutional layer receives the feature map output from the output of the 1st channel attention layer. The input of the 25th convolutional layer receives the feature map output from the output of the 24th convolutional layer. The input of the 22nd batch normalization layer receives the feature map output from the output of the 25th convolutional layer. The input of the 17th ReLU activation layer receives the feature map output from the output of the 22nd batch normalization layer. The input of the 26th convolutional layer receives the feature map output from the output of the 17th ReLU activation layer. The input of the 23rd batch normalization layer receives the feature map output from the output of the 26th convolutional layer. The input of the 18th ReLU activation layer receives the feature map output from the output of the 23rd batch normalization layer. The feature map output by the 18th ReLU activation layer is denoted as... The first input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform channel stitching and denote the resulting feature map as... The second channel of the attention layer receives input. The input of the 27th convolutional layer receives the feature map output from the output of the 2nd channel attention layer. The feature map output from the 27th convolutional layer is denoted as... The input of the 28th convolutional layer receives The input of the 19th ReLU activation layer receives the feature map output from the 28th convolutional layer, the input of the 29th convolutional layer receives the feature map output from the 19th ReLU activation layer, and the input of the 3rd Sigmoid activation layer receives the feature map output from the 29th convolutional layer. The feature map output from the 3rd Sigmoid activation layer is denoted as S. j S j That is, the feature map output by the output end of the j-th semantic enhancement decoding block; where 1≤j≤3, This represents the feature map output from the (j+2)th feature selection and fusion block. Convolutional layers 20 through 29 are all 2D convolutional layers. In the first semantic enhancement decoding block: the input size of the 20th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, and the dilation rate is 0. The input size of the 28th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information].

[0024] Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is In the second semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is In the third semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is

[0025] The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is

[0026] Compared with the prior art, the advantages of the present invention are as follows:

[0027] 1) The neural network constructed by the method of the present invention uses a ResNet50 backbone network for feature extraction, extracting features from color visible light images and infrared images; then, an asymmetric feature complementation module is used to allow full interaction between features of different modalities through an asymmetric strategy; then, a feature selection and fusion module is used to select and fuse valuable features in the channel and spatial dimensions; finally, three semantic enhancement decoding blocks are used to decode accurate salient target detection results.

[0028] 2) Based on the inherent differences and characteristics between the two modalities, the method of the present invention employs an asymmetric feature complementarity module (including a color visible light feature complementarity module and an infrared feature complementarity module) in the constructed neural network to allow the two modalities to interact fully while reducing the introduction of interference information, so as to fully interact with multimodal complementary information, reduce the modal differences between different modalities, and improve the accuracy of salient target detection in color visible light and infrared images.

[0029] 3) The method of this invention addresses the problem of high-quality fusion of multimodal features by employing a feature selection and fusion module in the constructed neural network. This module can filter out valuable features in the channel and spatial dimensions and designs a fusion strategy to achieve the fusion of color visible light features and infrared features to solve the problem of internal differences between different modalities, thereby effectively improving the accuracy of salient target detection in color visible light and infrared images. Attached Figure Description

[0030] Figure 1 This is a diagram illustrating the overall implementation framework of the method of the present invention;

[0031] Figure 2 A schematic diagram illustrating the structural composition of the neural network constructed using the method of this invention;

[0032] Figure 3 A schematic diagram of the composition structure of the complementary color visible light feature blocks in the neural network constructed by the method of the present invention;

[0033] Figure 4 A schematic diagram of the composition structure of the infrared feature complementary block in the neural network constructed by the method of the present invention;

[0034] Figure 5 A schematic diagram of part of the feature selection and fusion block in the neural network constructed by the method of the present invention;

[0035] Figure 6 A schematic diagram of the remaining structural components of the feature selection and fusion block in the neural network constructed by the method of the present invention;

[0036] Figure 7 This is a schematic diagram illustrating the composition structure of the semantic enhancement decoding block in the neural network constructed by the method of this invention. Detailed Implementation

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] The present invention proposes a method for salient target detection in color visible light and infrared images, the overall implementation framework of which is shown in the figure below. Figure 1As shown, the method first constructs a training set containing several pairs of color visible light images and their corresponding infrared images, and builds a neural network; secondly, it inputs the several pairs of color visible light images and their corresponding infrared images from the training set into the neural network for multiple rounds of network training, and obtains a neural network training model after the network training is completed; thirdly, it uses the neural network training model to predict the test image pairs, and predicts the salient target images of the test image pairs, characterized by: as Figure 2 As shown, the neural network mainly consists of a feature extraction module, an asymmetric feature complementation module, a feature selection and fusion module, and a semantic enhancement decoder, wherein:

[0039] The feature extraction module includes two ResNet50 backbone networks. The first layer of the first ResNet50 backbone network receives a color visible light image of size H×W×3 as its input. The feature map output from the first layer of the first ResNet50 backbone network is denoted as FR1. The second layer of the first ResNet50 backbone network receives FR1 as its input, and the feature map output from the second layer of the first ResNet50 backbone network is denoted as FR2. The third layer of the first ResNet50 backbone network... The input terminal receives FR2, and the feature map output from the third layer of the first ResNet50 backbone network is denoted as FR3. The input terminal of the fourth layer of the first ResNet50 backbone network receives FR3, and the feature map output from the fourth layer of the first ResNet50 backbone network is denoted as FR4. The input terminal of the fifth layer of the first ResNet50 backbone network receives FR4, and the feature map output from the fifth layer of the first ResNet50 backbone network is denoted as FR5. The second ResNet50 backbone network... The first layer of the second ResNet50 backbone network receives an infrared image of size H×W×3 as its input. The feature map output from the first layer of the second ResNet50 backbone network is denoted as FT1. The second layer of the second ResNet50 backbone network receives FT1 as its input, and the feature map output from the second layer of the second ResNet50 backbone network is denoted as FT2. The third layer of the second ResNet50 backbone network receives FT2 as its input, and the feature map output from the third layer of the second ResNet50 backbone network is denoted as... FT3 is the input of the fourth layer of the second ResNet50 backbone network, and FT4 is the output feature map of the fourth layer of the second ResNet50 backbone network. FT4 is the input of the fifth layer of the second ResNet50 backbone network, and FT5 is the output feature map of the fifth layer of the second ResNet50 backbone network. The ResNet50 backbone network is an existing structural framework, and its network structure has been publicly disclosed, as described in the literature K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. (“Image Recognition Based on Deep Residual Learning”). The ResNet50 backbone network has five layers, and the sizes of FR1 and FT1 are... The sizes of FR2 and FT2 are The sizes of FR3 and FT3 are The sizes of FR4 and FT4 are The sizes of FR5 and FT5 are In this embodiment, H×W is set to 256×256.

[0040] The asymmetric feature complementation module consists of a color visible light feature complementation module and an infrared feature complementation module. The color visible light feature complementation module comprises five structurally identical color visible light feature complementation blocks; the first input terminal of the first color visible light feature complementation block receives FR1, the second input terminal receives FT1, and the feature map output from the output terminal of the first color visible light feature complementation block is denoted as... The first input of the second color visible light feature complementary block receives FR2, the second input receives FT2, and the feature map output from the output of the second color visible light feature complementary block is denoted as follows. The first input of the third color visible light feature complement block receives FR3, the second input receives FT3, and the feature map output from the output of the third color visible light feature complement block is denoted as follows. The first input of the fourth color visible light feature complementary block receives FR4, the second input receives FT4, and the feature map output from the output of the fourth color visible light feature complementary block is denoted as follows. The first input of the fifth color visible light feature complementary block receives FR5, the second input receives FT5, and the feature map output from the output of the fifth color visible light feature complementary block is denoted as follows. in, The size is The size is The size is The size is The size is The infrared feature complementation module comprises five identical infrared feature complementation blocks; the first input terminal of the first infrared feature complementation block receives FT1, the second input terminal receives FR1, and the feature map output from the output terminal of the first infrared feature complementation block is denoted as... The first input terminal of the second infrared feature complement block receives FT2, the second input terminal receives FR2, and the feature map output by the output terminal of the second infrared feature complement block is denoted as... The first input terminal of the third infrared feature complement block receives FT3, the second input terminal receives FR3, and the feature map output by the output terminal of the third infrared feature complement block is denoted as... The first input terminal of the fourth infrared feature complement block receives FT4, the second input terminal receives FR4, and the feature map output from the output terminal of the fourth infrared feature complement block is denoted as follows. The first input terminal of the fifth infrared feature complementary block receives FT5, the second input terminal receives FR5, and the feature map output from the output terminal of the fifth infrared feature complementary block is denoted as... in, The size is The size is The size is The size is The size is

[0041] The feature selection and fusion module includes five structurally identical feature selection and fusion blocks; the first input of the first feature selection and fusion block receives... The second input terminal receives The feature map output from the first feature selection and fusion block is denoted as follows: The first input terminal of the second feature selection and fusion block receives... The second input terminal receives The feature map output from the output of the second feature selection and fusion block is denoted as follows: The first input terminal of the third feature selection and fusion block receives... The second input terminal receives The feature map output from the third feature selection and fusion block is denoted as follows: The first input terminal of the fourth feature selection and fusion block receives The second input terminal receives The feature map output from the output of the fourth feature selection and fusion block is denoted as follows: The first input terminal of the fifth feature selection and fusion block receives... The second input terminal receives The feature map output by the 5th feature selection and fusion block is denoted as follows: in, The size is The size is The size is The size is The size is

[0042] The semantic enhancement decoder consists of three structurally identical semantic enhancement decoding blocks; the first input of the first semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output by the output of the first semantic enhancement decoding block is denoted as S1, and S1 is used as the first coarse saliency target image; the first input of the second semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the output of the second semantic enhancement decoding block is denoted as S2, and S2 is used as the second coarse saliency target image; the first input of the third semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the third semantic enhancement decoding block is denoted as S3, and S3 is used as the third coarse saliency target image; where the size of S1 is... The size of S2 is The size of S3 is

[0043] The first coarse saliency target image S1, the second coarse saliency target image S2, and the third coarse saliency target image S3 are summed element-wise and then upsampled. The resulting feature map is denoted as S. f and S f The final salient target image; where element-wise addition is a common operation in neural networks, and the upsampling operation follows a bilinear pattern, S f Its size is H×W×1.

[0044] In one specific embodiment, the training set is constructed as follows: at least 200 pairs of original color visible light images and their corresponding original infrared images are selected; then, each original color visible light image and its corresponding original infrared image is downsampled to H×W; then, all color visible light images and their corresponding infrared images of size H×W are used to form the training set; wherein, in this embodiment, 2500 pairs of original color visible light images and their corresponding original infrared images are selected, and H=W=256.

[0045] In one specific embodiment, the process of obtaining the neural network training model is as follows: each pair of color visible light images and their corresponding infrared images in the training set are input into the neural network for network training. Before the end of each round of network training, a loss function L is calculated to optimize the neural network. The neural network training model is obtained after a total of 150 rounds of network training. Specifically, in the first 40 rounds of network training... In the last 110 rounds of network training S f Let G represent the final salient target image output by the neural network, and S represent the label image. j Let L represent the j-th coarse saliency target image obtained in the neural network. bce () represents the binary cross-entropy (BCE) loss, L iou () indicates the Intersection over Union (IoU) loss.

[0046] In a specific embodiment, the process of using a neural network training model to predict the salient target image of a test image pair is as follows: arbitrarily select a pair of original color visible light images and their corresponding original infrared images; then perform a downsampling operation on the pair of original color visible light images and original infrared images to downsample the image size to H×W, and use this as the test image pair; then input the test image pair into the neural network training model to predict the salient target image of the test image pair; where H=W=256.

[0047] The five complementary blocks of color visible light features have the same structure, only their inputs and outputs differ. In a specific embodiment, such as... Figure 3 As shown, the color visible light feature complementary block mainly consists of the first to fifth convolutional layers, the first to fifth batch normalization layers, and the first to fifth ReLU activation layers. For the i-th color visible light feature complementary block, the input of the first convolutional layer serves as the first input of the i-th color visible light feature complementary block, receiving FR. i The input of the first Batch Normalization layer receives the feature map output from the first convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first Batch Normalization layer. The feature map output from the first ReLU activation layer is denoted as... The input of the second convolutional layer serves as the second input of the i-th color visible light feature complementary block, receiving the FT. i The input of the second Batch Normalization layer receives the feature map output from the second convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second Batch Normalization layer. The feature map output from the second ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform a channel concatenation operation and denote the resulting feature map as... The input of the third convolutional layer receives The input of the third Batch Normalization layer receives the feature map output from the third convolutional layer, and the input of the third ReLU activation layer receives the feature map output from the third Batch Normalization layer. The feature map output from the third ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the fourth convolutional layer receives The input of the fourth Batch Normalization layer receives the feature map output from the fourth convolutional layer, and the input of the fourth ReLU activation layer receives the feature map output from the fourth Batch Normalization layer. The feature map output from the fourth ReLU activation layer is denoted as... The input of the 5th convolutional layer receives FR i The input of the 5th Batch Normalization layer receives the feature map output from the 5th convolutional layer, and the input of the 5th ReLU activation layer receives the feature map output from the 5th Batch Normalization layer. The feature map output from the 5th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th color visible light feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The first to fifth convolutional layers are all 2D convolutional layers. In the first color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the second color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the third color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 4th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 5th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the autofill is 1.

[0048] Here, element-wise multiplication, element-wise addition, element-wise subtraction, and concatenation are all common operations in neural networks; Figure 3 The term BN is short for Batch Normalization layer.

[0049] The five infrared feature complementary blocks have the same structure, only their inputs and outputs differ. In a specific embodiment, such as... Figure 4 As shown, the infrared feature complement block mainly consists of the 6th to 9th convolutional layers, the 6th to 9th batch normalization layers, the 6th to 9th ReLU activation layers, and the 1st spatial attention layer. For the i-th infrared feature complement block, the input of the 6th convolutional layer serves as the first input of the i-th infrared feature complement block to receive the FT. i The input of the 6th Batch Normalization layer receives the feature map output from the 6th convolutional layer, and the input of the 6th ReLU activation layer receives the feature map output from the 6th Batch Normalization layer. The feature map output from the 6th ReLU activation layer is denoted as FT. i 2_1 The input of the 7th convolutional layer serves as the second input of the i-th infrared feature complementary block, receiving FR. i The input of the 7th Batch Normalization layer receives the feature map output from the 7th convolutional layer, and the input of the 7th ReLU activation layer receives the feature map output from the 7th Batch Normalization layer. The feature map output from the 7th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform a channel concatenation operation and denote the resulting feature map as... The input of the 8th convolutional layer receives The input of the 8th Batch Normalization layer receives the feature map output from the 8th convolutional layer, and the input of the 8th ReLU activation layer receives the feature map output from the 8th Batch Normalization layer. The feature map output from the 8th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the 9th convolutional layer receives The input of the 9th Batch Normalization layer receives the feature map output from the 9th convolutional layer, and the input of the 9th ReLU activation layer receives the feature map output from the 9th Batch Normalization layer. The feature map output from the 9th ReLU activation layer is denoted as... The input of the first spatial attention layer receives FR i The feature map output by the first spatial attention layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th infrared feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The 6th to 9th convolutional layers are all 2D convolutional layers. In the first infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the second infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the third infrared feature complement block: the input size of the sixth and seventh convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 4th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 5th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the autofill is 1, and the dilation rate is 1.

[0050] Here, element-wise multiplication, element-wise addition, element-wise subtraction, and concatenation are all common operations in neural networks; Figure 4 In this context, BN layer is short for Batch Normalization layer. Spatial attention layer is an existing module in neural networks, publicly used in many papers, such as S.Woo, J.Park, J.-Y.Lee, and ISKweon, “CBAM: Convolutional block attention module,” in Proc. of European Conference on Computer Vision, pp. 3-19, Sep. 2018.

[0051] The five feature selections have the same structure as the fusion block, only the inputs and outputs are different. In a specific embodiment, such as... Figure 5 and Figure 6 As shown, the feature selection and fusion block mainly consists of the first average pooling layer, the second average pooling layer, the first max pooling layer, the second max pooling layer, the first fully connected layer, the second fully connected layer, the first sigmoid activation layer, the second sigmoid activation layer, the 10th to 19th convolutional layers, the 10th to 17th batch normalization layers, the 10th to 12th ReLU activation layers, and the first to 6th PReLU activation layers. For the i-th feature selection and fusion block, the inputs of the first average pooling layer and the first max pooling layer are both received as the first input of the i-th feature selection and fusion block. The feature map output from the first average pooling layer is denoted as... The feature map output from the first max pooling layer is denoted as follows: The input terminals of the second average pooling layer and the second max pooling layer are both received as the second input terminal of the i-th feature selection and fusion block. The feature map output from the second max-pooling layer is denoted as... The feature map output from the output of the second average pooling layer is denoted as... right and Perform a channel concatenation operation and denote the resulting feature map as... right and Perform a channel concatenation operation and denote the resulting feature map as... The input of the first fully connected layer receives... The feature map output from the first fully connected layer is denoted as... The input of the second fully connected layer receives... The feature map output from the output of the second fully connected layer is denoted as... right and Perform element-wise addition and use the resulting feature map as input to the first Sigmoid activation layer. Then, perform channel segmentation on the feature map output from the first Sigmoid activation layer and denote the resulting feature maps as follows: and right and Perform channel-by-channel multiplication and denote the resulting feature map as... right and Perform channel-by-channel multiplication and denote the resulting feature map as... The input of the 10th convolutional layer receives The input of the 10th Batch Normalization layer receives the feature map output from the 10th convolutional layer, and the input of the 1st PReLU activation layer receives the feature map output from the 10th Batch Normalization layer. The feature map output from the 1st PReLU activation layer is denoted as... The input of the 11th convolutional layer receives The input of the 11th Batch Normalization layer receives the feature map output from the 11th convolutional layer, and the input of the 2nd PReLU activation layer receives the feature map output from the 11th Batch Normalization layer. The feature map output from the 2nd PReLU activation layer is denoted as... The input of the 12th convolutional layer receives The input of the 12th Batch Normalization layer receives the feature map output from the 12th convolutional layer, and the input of the 3rd PReLU activation layer receives the feature map output from the 12th Batch Normalization layer. The feature map output from the 3rd PReLU activation layer is denoted as... The input of the 13th convolutional layer receives The input of the 13th Batch Normalization layer receives the feature map output from the 13th convolutional layer, and the input of the 4th PReLU activation layer receives the feature map output from the 13th Batch Normalization layer. The feature map output from the 4th PReLU activation layer is denoted as... The input of the 14th convolutional layer receives The input of the 14th Batch Normalization layer receives the feature map output from the 14th convolutional layer, and the input of the 5th PReLU activation layer receives the feature map output from the 14th Batch Normalization layer. The feature map output from the 5th PReLU activation layer is denoted as... The input of the 15th convolutional layer receives The input of the 15th Batch Normalization layer receives the feature map output from the 15th convolutional layer, and the input of the 6th PReLU activation layer receives the feature map output from the 15th Batch Normalization layer. The feature map output from the 6th PReLU activation layer is denoted as... right and Perform a channel concatenation operation and denote the resulting feature map as... right and Perform a channel concatenation operation and denote the resulting feature map as... The input of the 16th convolutional layer receives The input of the 16th Batch Normalization layer receives the feature map output from the 16th convolutional layer, and the input of the 10th ReLU activation layer receives the feature map output from the 16th Batch Normalization layer. The feature map output from the 10th ReLU activation layer is denoted as... The input of the 17th convolutional layer receives The input of the 17th Batch Normalization layer receives the feature map output from the 17th convolutional layer, and the input of the 11th ReLU activation layer receives the feature map output from the 17th Batch Normalization layer. The feature map output from the 11th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right Perform a channel concatenation operation and denote the resulting feature map as... The input of the 18th convolutional layer receives The input of the 12th ReLU activation layer receives the feature map output from the 18th convolutional layer, and the input of the 19th convolutional layer receives the feature map output from the 12th ReLU activation layer. The feature map output from the 19th convolutional layer is denoted as... The input of the second Sigmoid activation layer receives The feature maps obtained by channel segmentation of the feature map output from the second Sigmoid activation layer are denoted as follows: and right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output by the i-th feature selection and fusion block; where 1 ≤ i ≤ 5. This represents the feature map output from the output terminal of the i-th infrared feature complementary block. This represents the feature map output from the output of the i-th color visible light feature complementary block. The 10th to 19th convolutional layers are all 2D convolutional layers. In the first feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the second feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×512, the size of the intermediate hidden layers is 1×1×256, and the output size is 1×1×512. and The size is 1×1×256, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the third feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the fourth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the fifth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is

[0052] Here, channel concatenation, element-wise addition, element-wise multiplication, element-wise subtraction, channel-wise multiplication, and channel splitting are all common operations in neural networks. Channel splitting involves splitting a 3D feature map into two feature maps along the channel dimension. Their height and width are the same as the original feature map, and the sum of their channel numbers equals the number of channels in the original feature map. In channel-wise multiplication, the two feature maps have different widths and heights but the same number of channels. One of the feature maps has a width and height of 1. Channel-wise multiplication is equivalent to copying this feature map to each channel, making it the same size as the other feature map, and then performing element-wise multiplication. Figure 6 The BN layer is short for BatchNormalization layer.

[0053] The three semantic enhancement decoding blocks have the same structure, differing only in their inputs and outputs. In a specific embodiment, such as... Figure 7 As shown, the semantic enhancement decoding block mainly consists of convolutional layers 20 to 29, batch normalization layers 18 to 23, ReLU activation layers 13 to 19, sigmoid activation layer 3, a Transformer layer, a spatial attention layer 2, a channel attention layer 1, a channel attention layer 2, and upsampling layers 1 to 3. For the j-th semantic enhancement decoding block, the input of the 20th convolutional layer is used as the first input of the j-th semantic enhancement decoding block. The input of the 18th Batch Normalization layer receives the feature map output from the 20th convolutional layer; the input of the 13th ReLU activation layer receives the feature map output from the 18th Batch Normalization layer; the input of the 1st upsampling layer receives the feature map output from the 13th ReLU activation layer; the input of the 21st convolutional layer receives the feature map output from the 1st upsampling layer; the input of the 19th Batch Normalization layer receives the feature map output from the 21st convolutional layer; the input of the 14th ReLU activation layer receives the feature map output from the 19th Batch Normalization layer; the input of the Transformer layer receives the feature map output from the 14th ReLU activation layer; and the feature map output by the Transformer layer is denoted as... The input terminal of the second spatial attention layer receives The input of the second upsampling layer receives the feature map output from the output of the second spatial attention layer. The feature map output from the second upsampling layer is denoted as... The second input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... The input of the 22nd convolutional layer receives The input of the 20th Batch Normalization layer receives the feature map output from the 22nd convolutional layer. The input of the 15th ReLU activation layer receives the feature map output from the 20th Batch Normalization layer. The input of the 3rd upsampling layer receives the feature map output from the 15th ReLU activation layer. The input of the 23rd convolutional layer receives the feature map output from the 3rd upsampling layer. The input of the 21st Batch Normalization layer receives the feature map output from the 23rd convolutional layer. The input of the 16th ReLU activation layer receives the feature map output from the 21st Batch Normalization layer. The feature map output from the 16th ReLU activation layer is denoted as... right and Perform channel stitching and denote the resulting feature map as... The first channel of the attention layer receives input. The input of the 24th convolutional layer receives the feature map output from the output of the 1st channel attention layer. The input of the 25th convolutional layer receives the feature map output from the output of the 24th convolutional layer. The input of the 22nd batch normalization layer receives the feature map output from the output of the 25th convolutional layer. The input of the 17th ReLU activation layer receives the feature map output from the output of the 22nd batch normalization layer. The input of the 26th convolutional layer receives the feature map output from the output of the 17th ReLU activation layer. The input of the 23rd batch normalization layer receives the feature map output from the output of the 26th convolutional layer. The input of the 18th ReLU activation layer receives the feature map output from the output of the 23rd batch normalization layer. The feature map output by the 18th ReLU activation layer is denoted as... The first input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform channel stitching and denote the resulting feature map as... The second channel of the attention layer receives input. The input of the 27th convolutional layer receives the feature map output from the output of the 2nd channel attention layer. The feature map output from the 27th convolutional layer is denoted as... The input of the 28th convolutional layer receives The input of the 19th ReLU activation layer receives the feature map output from the 28th convolutional layer, the input of the 29th convolutional layer receives the feature map output from the 19th ReLU activation layer, and the input of the 3rd Sigmoid activation layer receives the feature map output from the 29th convolutional layer. The feature map output from the 3rd Sigmoid activation layer is denoted as S. j S j That is, the feature map output by the output end of the j-th semantic enhancement decoding block; where 1≤j≤3, This represents the feature map output from the (j+2)th feature selection and fusion block. Convolutional layers 20 through 29 are all 2D convolutional layers. In the first semantic enhancement decoding block: the input size of the 20th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and the interpolation mode set to bilinear interpolation, the input size of the Transformer layer is... Output size is In the second semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and the interpolation mode set to bilinear interpolation, the input size of the Transformer layer is... Output size is In the third semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, and the dilation rate is 0. The input size of the 28th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and the interpolation mode set to bilinear interpolation, the input size of the Transformer layer is... Output size is

[0054] Here, element-wise multiplication and channel concatenation are common operations in neural networks; Figure 7In this context, BN layer is short for Batch Normalization layer. Spatial attention layer and channel attention layer are existing modules in neural networks, publicly used in many papers, such as S. Woo, J. Park, J.-Y. Lee, and ISKweon, “CBAM: Convolutional block attention module,” in Proc. of European Conference on Computer Vision, pp. 3-19, Sep. 2018. Similarly, Transformer layer is an existing module in neural networks, publicly used in many papers, such as Z. Liu, Y. Wang, Z. Tu, Y. Xiao, and B. Tang, “TriTransNet: RGB-D salient object detection with atriplet transformer embedding network,” arXiv preprint arXiv:2108.03990, 2021.

[0055] To further demonstrate the feasibility and effectiveness of the method of the present invention, experiments were conducted on the method of the present invention.

[0056] In this embodiment, the method of the present invention is used to test the salient target detection dataset (VT5000) of color visible light and infrared images established by Anhui University. The training set in VT5000 includes 2500 pairs of color visible light images and infrared images, and the test set in VT5000 contains a total of 2500 pairs of color visible light images and infrared images, i.e., 2500 pairs of test images.

[0057] In this embodiment, four commonly used objective parameters are selected to evaluate the performance of the method of the present invention: S-measure, E-measure, F-measure, and Mean Absolute Error (MAE). Table 1 shows the correlation between salient target images and label images obtained by the method of the present invention on the VT5000 dataset.

[0058] Table 1 shows the S-measure, E-measure, F-measure, and Mean Absolute Error (MAE) between salient target images and label images obtained using the method of this invention on the VT5000 dataset.

[0059] S-measure E-measure F-measure MAE Method of the present invention 0.899 0.943 0.873 0.027

[0060] The results shown in Table 1 indicate that the method of the present invention achieves high S-measure, E-measure, F-measure and low MAE on the existing color visible light and infrared image salient target detection dataset. This shows that the salient target images and label images obtained by the method of the present invention are relatively close, and the method of the present invention can effectively complete the salient target detection of color visible light and infrared images.

Claims

1. A method for detecting salient targets in color visible light and infrared images, comprising: firstly, constructing a training set containing several pairs of color visible light images and their corresponding infrared images, and building a neural network; secondly, inputting the several pairs of color visible light images and their corresponding infrared images from the training set into the neural network for multiple rounds of network training, obtaining a neural network training model after the network training is completed; and thirdly, using the neural network training model to predict test image pairs, thereby predicting the salient target images of the test image pairs, characterized in that: The neural network mainly consists of a feature extraction module, an asymmetric feature complementation module, a feature selection and fusion module, and a semantic enhancement decoder, wherein: The feature extraction module includes two ResNet50 backbone networks; the input of the first layer of the first ResNet50 backbone network receives a color visible light image of size H×W×3, the output of the first layer of the first ResNet50 backbone network is denoted as FR1, the input of the second layer of the first ResNet50 backbone network receives FR1, the output of the second layer of the first ResNet50 backbone network is denoted as FR2, the input of the third layer of the first ResNet50 backbone network receives FR2, the output of the third layer of the first ResNet50 backbone network is denoted as FR3, the input of the fourth layer of the first ResNet50 backbone network receives FR3, the output of the fourth layer of the first ResNet50 backbone network is denoted as FR4, the input of the fifth layer of the first ResNet50 backbone network receives FR4, and the output of the fifth layer of the first ResNet50 backbone network is denoted as FR5; The first layer of two ResNet50 backbone networks receives an infrared image of size H×W×3 as its input. The feature map output from the first layer of the second ResNet50 backbone network is denoted as FT1. The second layer of the second ResNet50 backbone network receives FT1 as its input, and the feature map output from the second layer of the second ResNet50 backbone network is denoted as FT2. The third layer of the second ResNet50 backbone network receives FT2 as its input. The feature map output from the third layer of the network is denoted as FT3. The input of the fourth layer of the second ResNet50 backbone network receives FT3, and the output of the fourth layer of the second ResNet50 backbone network is denoted as FT4. The input of the fifth layer of the second ResNet50 backbone network receives FT4, and the output of the fifth layer of the second ResNet50 backbone network is denoted as FT5. The ResNet50 backbone network has five layers. The sizes of FR1 and FT1 are... The sizes of FR2 and FT2 are The sizes of FR3 and FT3 are The sizes of FR4 and FT4 are The sizes of FR5 and FT5 are The asymmetric feature complementation module is composed of a color visible light feature complementation module and an infrared feature complementation module; The aforementioned color visible light feature complementation module comprises five structurally identical color visible light feature complementation blocks. Each color visible light feature complementation block mainly consists of convolutional layers 1 to 5, batch normalization layers 1 to 5, and ReLU activation layers 1 to 5. The first input of the first color visible light feature complementation block receives FR1, the second input receives FT1, and the feature map output from the first color visible light feature complementation block is denoted as... The first input of the second color visible light feature complementary block receives FR2, the second input receives FT2, and the feature map output from the output of the second color visible light feature complementary block is denoted as follows. The first input of the third color visible light feature complement block receives FR3, the second input receives FT3, and the feature map output from the output of the third color visible light feature complement block is denoted as follows. The first input of the fourth color visible light feature complementary block receives FR4, the second input receives FT4, and the feature map output from the output of the fourth color visible light feature complementary block is denoted as follows. The first input of the fifth color visible light feature complementary block receives FR5, the second input receives FT5, and the feature map output from the output of the fifth color visible light feature complementary block is denoted as follows. in, The size is The size is The size is The size is The size is The infrared feature complementation module comprises five identical infrared feature complementation blocks. Each infrared feature complementation block mainly consists of the 6th to 9th convolutional layers, the 6th to 9th batch normalization layers, the 6th to 9th ReLU activation layers, and the 1st spatial attention layer. The first input of the 1st infrared feature complementation block receives FT1, the second input receives FR1, and the feature map output from the 1st infrared feature complementation block is denoted as... The first input terminal of the second infrared feature complement block receives FT2, the second input terminal receives FR2, and the feature map output by the output terminal of the second infrared feature complement block is denoted as... The first input terminal of the third infrared feature complement block receives FT3, the second input terminal receives FR3, and the feature map output by the output terminal of the third infrared feature complement block is denoted as... The first input terminal of the fourth infrared feature complement block receives FT4, the second input terminal receives FR4, and the feature map output from the output terminal of the fourth infrared feature complement block is denoted as follows. The first input terminal of the fifth infrared feature complementary block receives FT5, the second input terminal receives FR5, and the feature map output from the output terminal of the fifth infrared feature complementary block is denoted as... in, The size is The size is The size is The size is The size is The feature selection and fusion module includes five structurally identical feature selection and fusion blocks. Each feature selection and fusion block mainly consists of a first average pooling layer, a second average pooling layer, a first max pooling layer, a second max pooling layer, a first fully connected layer, a second fully connected layer, a first sigmoid activation layer, a second sigmoid activation layer, 10th to 19th convolutional layers, 10th to 17th batch normalization layers, 10th to 12th ReLU activation layers, and 1st to 6th PReLU activation layers. The first input of the first feature selection and fusion block receives... The second input terminal receives The feature map output from the first feature selection and fusion block is denoted as follows: The first input terminal of the second feature selection and fusion block receives... The second input terminal receives The feature map output from the output of the second feature selection and fusion block is denoted as follows: The first input terminal of the third feature selection and fusion block receives... The second input terminal receives The feature map output from the third feature selection and fusion block is denoted as follows: The first input terminal of the fourth feature selection and fusion block receives The second input terminal receives The feature map output from the output of the fourth feature selection and fusion block is denoted as follows: The first input terminal of the fifth feature selection and fusion block receives... The second input terminal receives The feature map output by the 5th feature selection and fusion block is denoted as follows: in, The size is The size is The size is The size is The size is The semantic enhancement decoder comprises three structurally identical semantic enhancement decoding blocks. Each semantic enhancement decoding block mainly consists of convolutional layers 20 to 29, batch normalization layers 18 to 23, ReLU activation layers 13 to 19, a sigmoid activation layer 3, a Transformer layer, a spatial attention layer 2, a channel attention layer 1, a channel attention layer 2, and upsampling layers 1 to 3. The first input of the first semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output by the output of the first semantic enhancement decoding block is denoted as S1, and S1 is used as the first coarse saliency target image; the first input of the second semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the output of the second semantic enhancement decoding block is denoted as S2, and S2 is used as the second coarse saliency target image; the first input of the third semantic enhancement decoding block receives... The second input terminal receives The third input terminal receives The feature map output from the third semantic enhancement decoding block is denoted as S3, and S3 is used as the third coarse saliency target image; where the size of S1 is... The size of S2 is The size of S3 is The first coarse saliency target image S1, the second coarse saliency target image S2, and the third coarse saliency target image S3 are summed element-wise and then upsampled. The resulting feature map is denoted as S. f and S f As the final salient target image; where the upsampling operation mode is bilinear, S f The size is H×W×1; The process of obtaining the neural network training model is as follows: Each pair of color visible light images and their corresponding infrared images from the training set are input into the neural network for training. Before the end of each training round, the loss function L is calculated to optimize the neural network. After a total of 150 training rounds, the neural network training model is obtained. Specifically, in the first 40 rounds of training… In the last 110 rounds of network training S f Let G represent the final salient target image output by the neural network, and S represent the label image. j Let L represent the j-th coarse saliency target image obtained in the neural network. bce () represents the binary cross-entropy loss, L iou () indicates the intersection and union ratio loss.

2. The method for detecting salient targets in color visible light and infrared images according to claim 1, characterized in that: The training set is constructed as follows: select at least 200 pairs of original color visible light images and their corresponding original infrared images; then perform downsampling operation on each original color visible light image and its corresponding original infrared image to downsample the image size to H×W; and then construct the training set from all the color visible light images and their corresponding infrared images of size H×W.

3. The method for detecting salient targets in color visible light and infrared images according to claim 1, characterized in that: The process of using a neural network training model to predict the salient target image of a test image pair is as follows: arbitrarily select a pair of original color visible light images and their corresponding original infrared images; then downsample the pair of original color visible light images and original infrared images to reduce the image size to H×W, and use them as the test image pair; then input the test image pair into the neural network training model to predict the salient target image of the test image pair.

4. The method for detecting salient targets in color visible light and infrared images according to any one of claims 1 to 3, characterized in that: For the i-th color visible light feature complementary block, the input of the first convolutional layer serves as the first input of the i-th color visible light feature complementary block, receiving FR. i The input of the first Batch Normalization layer receives the feature map output from the first convolutional layer, and the input of the first ReLU activation layer receives the feature map output from the first Batch Normalization layer. The feature map output from the first ReLU activation layer is denoted as... The input of the second convolutional layer serves as the second input of the i-th color visible light feature complementary block, receiving the FT. i The input of the second Batch Normalization layer receives the feature map output from the second convolutional layer, and the input of the second ReLU activation layer receives the feature map output from the second Batch Normalization layer. The feature map output from the second ReLU activation layer is denoted as FT. i 1_1 ,right and FT i 1_1 Perform element-wise multiplication and denote the resulting feature map as... For FT i 1_1 and Perform element-wise subtraction, where FT i 1_1 The feature to be subtracted is denoted as FT. i 1_2 ,right and FT i 1_2 Perform channel connection operations and denote the resulting feature map as... The input of the third convolutional layer receives The input of the third Batch Normalization layer receives the feature map output from the third convolutional layer, and the input of the third ReLU activation layer receives the feature map output from the third Batch Normalization layer. The feature map output from the third ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the fourth convolutional layer receives The input of the fourth Batch Normalization layer receives the feature map output from the fourth convolutional layer, and the input of the fourth ReLU activation layer receives the feature map output from the fourth Batch Normalization layer. The feature map output from the fourth ReLU activation layer is denoted as... The input of the 5th convolutional layer receives FR i The input of the 5th Batch Normalization layer receives the feature map output from the 5th convolutional layer, and the input of the 5th ReLU activation layer receives the feature map output from the 5th Batch Normalization layer. The feature map output from the 5th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th color visible light feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The first to fifth convolutional layers are all 2D convolutional layers. In the first color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the second color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the third color visible light feature complementary block: the input size of the first and second convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 4th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The convolutional kernel size is 3×3, the stride is 1, and the auto-fill is 1; in the 5th color visible light feature complementary block: the input size of the 1st and 2nd convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the third convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 4th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the auto-padding is 1. The input size of the 5th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, and the autofill is 1.

5. The method for detecting salient targets in color visible light and infrared images according to any one of claims 1 to 3, characterized in that: For the i-th infrared feature complementary block, the input of the 6th convolutional layer serves as the first input of the i-th infrared feature complementary block to receive the FT. i The input of the 6th Batch Normalization layer receives the feature map output from the 6th convolutional layer, and the input of the 6th ReLU activation layer receives the feature map output from the 6th Batch Normalization layer. The feature map output from the 6th ReLU activation layer is denoted as FT. i 2_1 The input of the 7th convolutional layer serves as the second input of the i-th infrared feature complementary block, receiving FR. i The input of the 7th Batch Normalization layer receives the feature map output from the 7th convolutional layer, and the input of the 7th ReLU activation layer receives the feature map output from the 7th Batch Normalization layer. The feature map output from the 7th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform channel connection operations and denote the resulting feature map as... The input of the 8th convolutional layer receives The input of the 8th Batch Normalization layer receives the feature map output from the 8th convolutional layer, and the input of the 8th ReLU activation layer receives the feature map output from the 8th Batch Normalization layer. The feature map output from the 8th ReLU activation layer is denoted as... right and Perform element-wise addition and denote the resulting feature map as... The input of the 9th convolutional layer receives The input of the 9th Batch Normalization layer receives the feature map output from the 9th convolutional layer, and the input of the 9th ReLU activation layer receives the feature map output from the 9th Batch Normalization layer. The feature map output from the 9th ReLU activation layer is denoted as... The input of the first spatial attention layer receives FR i The feature map output by the first spatial attention layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... This refers to the feature map output from the output terminal of the i-th infrared feature complementary block; where 1≤i≤5, FR i This represents the feature map output from the i-th layer of the first ResNet50 backbone network, FT. i This represents the feature map output from the i-th layer of the second ResNet50 backbone network. The 6th to 9th convolutional layers are all 2D convolutional layers. In the first infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the second infrared feature complement block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the third infrared feature complement block: the input size of the sixth and seventh convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 4th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The convolutional kernel size is 3×3, the stride is 1, the auto-fill is 1, and the dilation rate is 1; in the 5th infrared feature complementary block: the input size of the 6th and 7th convolutional layers is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 8th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 9th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the autofill is 1, and the dilation rate is 1.

6. The method for detecting salient targets in color visible light and infrared images according to any one of claims 1 to 3, characterized in that: For the i-th feature selection and fusion block, the inputs of the first average pooling layer and the first max pooling layer are both received as the first input of the i-th feature selection and fusion block. The feature map output from the first average pooling layer is denoted as... The feature map output from the first max pooling layer is denoted as follows: The input terminals of the second average pooling layer and the second max pooling layer are both received as the second input terminal of the i-th feature selection and fusion block. The feature map output from the second max-pooling layer is denoted as... The feature map output from the output of the second average pooling layer is denoted as... right and Perform channel connection operations and denote the resulting feature map as... right and Perform channel connection operations and denote the resulting feature map as... The input of the first fully connected layer receives... The feature map output from the first fully connected layer is denoted as... The input of the second fully connected layer receives... The feature map output from the output of the second fully connected layer is denoted as... right and Perform element-wise addition and use the resulting feature map as input to the first Sigmoid activation layer. Then, perform channel segmentation on the feature map output from the first Sigmoid activation layer and denote the resulting feature maps as follows: and right and Perform channel-by-channel multiplication and denote the resulting feature map as... right and Perform channel-by-channel multiplication and denote the resulting feature map as... The input of the 10th convolutional layer receives The input of the 10th Batch Normalization layer receives the feature map output from the 10th convolutional layer, and the input of the 1st PReLU activation layer receives the feature map output from the 10th Batch Normalization layer. The feature map output from the 1st PReLU activation layer is denoted as... The input of the 11th convolutional layer receives The input of the 11th Batch Normalization layer receives the feature map output from the 11th convolutional layer, and the input of the 2nd PReLU activation layer receives the feature map output from the 11th Batch Normalization layer. The feature map output from the 2nd PReLU activation layer is denoted as... The input of the 12th convolutional layer receives The input of the 12th Batch Normalization layer receives the feature map output from the 12th convolutional layer, and the input of the 3rd PReLU activation layer receives the feature map output from the 12th Batch Normalization layer. The feature map output from the 3rd PReLU activation layer is denoted as... The input of the 13th convolutional layer receives The input of the 13th Batch Normalization layer receives the feature map output from the 13th convolutional layer, and the input of the 4th PReLU activation layer receives the feature map output from the 13th Batch Normalization layer. The feature map output from the 4th PReLU activation layer is denoted as... The input of the 14th convolutional layer receives The input of the 14th Batch Normalization layer receives the feature map output from the 14th convolutional layer, and the input of the 5th PReLU activation layer receives the feature map output from the 14th Batch Normalization layer. The feature map output from the 5th PReLU activation layer is denoted as... The input of the 15th convolutional layer receives The input of the 15th Batch Normalization layer receives the feature map output from the 15th convolutional layer, and the input of the 6th PReLU activation layer receives the feature map output from the 15th Batch Normalization layer. The feature map output from the 6th PReLU activation layer is denoted as... right and Perform channel connection operations and denote the resulting feature map as... right and Perform channel connection operations and denote the resulting feature map as... The input of the 16th convolutional layer receives The input of the 16th Batch Normalization layer receives the feature map output from the 16th convolutional layer, and the input of the 10th ReLU activation layer receives the feature map output from the 16th Batch Normalization layer. The feature map output from the 10th ReLU activation layer is denoted as... The input of the 17th convolutional layer receives The input of the 17th Batch Normalization layer receives the feature map output from the 17th convolutional layer, and the input of the 11th ReLU activation layer receives the feature map output from the 17th Batch Normalization layer. The feature map output from the 11th ReLU activation layer is denoted as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right and Perform element-wise subtraction, where As the feature to be subtracted, the resulting feature map is denoted as... right Perform channel connection operations and denote the resulting feature map as... The input of the 18th convolutional layer receives The input of the 12th ReLU activation layer receives the feature map output from the 18th convolutional layer, and the input of the 19th convolutional layer receives the feature map output from the 12th ReLU activation layer. The feature map output from the 19th convolutional layer is denoted as... The input of the second Sigmoid activation layer receives The feature maps obtained by channel segmentation of the feature map output from the second Sigmoid activation layer are denoted as follows: and right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform element-wise addition and denote the resulting feature map as... This refers to the feature map output by the i-th feature selection and fusion block; where 1 ≤ i ≤ 5. This represents the feature map output from the output terminal of the i-th infrared feature complementary block. This represents the feature map output from the output of the i-th color visible light feature complementary block. The 10th to 19th convolutional layers are all 2D convolutional layers. In the first feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the second feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×512, the size of the intermediate hidden layers is 1×1×256, and the output size is 1×1×512. and The size is 1×1×256, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the third feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the fourth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is In the fifth feature selection and fusion block: the input size of the first and second fully connected layers is 1×1×128, the size of the intermediate hidden layers is 1×1×64, and the output size is 1×1×128. and The size is 1×1×64, and the input size of the 10th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 11th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 12th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 13th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 14th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 2, the dilation rate is 2, and the input size of the 15th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 4, the dilation rate is 4, and the input size of the 16th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 17th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 18th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 19th convolutional layer is [missing information]. Output size is The convolution kernel size is 1×1, the stride is 1, the auto-fill is 0, and the dilation rate is 1. and The size is 7. The method for detecting salient targets in color visible light and infrared images according to any one of claims 1 to 3, characterized in that: For the j-th semantic enhancement decoding block, the input of the 20th convolutional layer is received as the first input of the j-th semantic enhancement decoding block. The input of the 18th Batch Normalization layer receives the feature map output from the 20th convolutional layer; the input of the 13th ReLU activation layer receives the feature map output from the 18th Batch Normalization layer; the input of the 1st upsampling layer receives the feature map output from the 13th ReLU activation layer; the input of the 21st convolutional layer receives the feature map output from the 1st upsampling layer; the input of the 19th Batch Normalization layer receives the feature map output from the 21st convolutional layer; the input of the 14th ReLU activation layer receives the feature map output from the 19th Batch Normalization layer; the input of the Transformer layer receives the feature map output from the 14th ReLU activation layer; and the feature map output by the Transformer layer is denoted as... The input terminal of the second spatial attention layer receives The input of the second upsampling layer receives the feature map output from the output of the second spatial attention layer. The feature map output from the second upsampling layer is denoted as... The second input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... The input of the 22nd convolutional layer receives The input of the 20th Batch Normalization layer receives the feature map output from the 22nd convolutional layer; the input of the 15th ReLU activation layer receives the feature map output from the 20th Batch Normalization layer; the input of the 3rd upsampling layer receives the feature map output from the 15th ReLU activation layer; the input of the 23rd convolutional layer receives the feature map output from the 3rd upsampling layer; the input of the 21st Batch Normalization layer receives the feature map output from the 23rd convolutional layer; and the input of the 16th ReLU activation layer receives the feature map output from the 21st Batch Normalization layer. The feature map output from the 16th ReLU activation layer is denoted as... right and Perform channel stitching and denote the resulting feature map as... The first channel of the attention layer receives input. The input of the 24th convolutional layer receives the feature map output from the output of the 1st channel attention layer. The input of the 25th convolutional layer receives the feature map output from the output of the 24th convolutional layer. The input of the 22nd Batch Normalization layer receives the feature map output from the output of the 25th convolutional layer. The input of the 17th ReLU activation layer receives the feature map output from the output of the 22nd Batch Normalization layer. The input of the 26th convolutional layer receives the feature map output from the output of the 17th ReLU activation layer. The input of the 23rd Batch Normalization layer receives the feature map output from the output of the 26th convolutional layer. The input of the 18th ReLU activation layer receives the feature map output from the output of the 23rd Batch Normalization layer. The feature map output by the 18th ReLU activation layer is denoted as... The first input of the j-th semantic enhancement decoding block receives right and Perform element-wise multiplication and denote the resulting feature map as... right and Perform channel stitching and denote the resulting feature map as... The second channel of the attention layer receives input. The input of the 27th convolutional layer receives the feature map output from the output of the 2nd channel attention layer. The feature map output from the 27th convolutional layer is denoted as... The input of the 28th convolutional layer receives The input of the 19th ReLU activation layer receives the feature map output from the 28th convolutional layer, the input of the 29th convolutional layer receives the feature map output from the 19th ReLU activation layer, and the input of the 3rd Sigmoid activation layer receives the feature map output from the 29th convolutional layer. The feature map output from the 3rd Sigmoid activation layer is denoted as S. j S j That is, the feature map output by the output end of the j-th semantic enhancement decoding block; where 1≤j≤3, This represents the feature map output from the (j+2)th feature selection and fusion block. Convolutional layers 20 through 29 are all 2D convolutional layers. In the first semantic enhancement decoding block: the input size of the 20th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is In the second semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 0, and the input size of the 28th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is In the third semantic enhancement decoding block: the input size of the 20th convolutional layer is Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 21st convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 22nd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 23rd convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 24th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 25th convolutional layer is [missing information]. Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 26th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, the dilation rate is 1, and the input size of the 27th convolutional layer is [missing information]. Output size is The kernel size is 1×1, the stride is 1, the auto-padding is 0, and the dilation rate is 0. The input size of the 28th convolutional layer is... Output size is The kernel size is 3×3, the stride is 1, the auto-padding is 1, the dilation rate is 1, and the input size of the 29th convolutional layer is [missing information]. Output size is The convolution kernel size is 3×3, the stride is 1, the auto-fill is 1, the dilation rate is 1, and the output feature size of the first upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the second upsampling layer is fixed. With the number of channels remaining constant and the mode set to bilinear interpolation, the output feature size of the third upsampling layer is fixed. With the number of channels remaining constant and bilinear interpolation mode used, the input size of the Transformer layer is... Output size is

Images