A salient object detection method based on infrared and visible image fusion

Abstract

The present application belongs to the technical field of target detection, and particularly relates to a salient target detection method based on infrared and visible light image fusion. The method comprises: inputting an infrared and visible light image pair corresponding to each other as a data set, and pre-processing the training set image in the data set to obtain a pre-processed training set; constructing a multi-modal salient target detection network; setting a loss function, and training the multi-modal salient target detection network according to the loss function and the pre-processed training set to obtain a multi-modal salient target detection model; and inputting an arbitrary infrared and visible light image pair into the multi-modal salient target detection model to obtain a salient target detection result. The method dynamically measures the feature effects of infrared and visible light modalities, fully considers the information between infrared and visible light modalities and the differences of salient regions during feature extraction, and refines and enhances the fused features, so that the salient target detection performance is better.

Description

Technical Field

[0001] This invention belongs to the fields of image processing and target detection technology, and specifically relates to a salient target detection method based on the fusion of infrared and visible light images. Background Technology

[0002] In computer vision tasks, the training scenarios used are typically idealized. Therefore, the trained models are often severely affected by unfavorable environmental factors such as uneven lighting, light intensity, visibility range, and severe occlusion. In such scenarios, visible light images suffer from low contrast, loss of detail, and missing usable information. The main purpose of using multimodal data is to introduce information from other modalities less affected by environmental interference, building upon the contrast, texture, and detail information of visible light images. In environments where a single visible light image cannot guarantee model performance, information from other modalities ensures sufficient scene information, enabling the model to be robust in complex environments.

[0003] Benefiting from equipment development and price reductions, depth and visible light, and infrared and visible light are two common combinations of multimodal data. Existing research shows that using depth and visible light for computer vision tasks significantly improves performance. However, in scenarios where depth information remains unchanged or changes drastically, the evaluation accuracy of depth images drops significantly, leading to reduced performance in those scenarios. In contrast, infrared images, through spectral filtering, capture only the corresponding wavelengths of light, enabling more accurate acquisition of scene infrared information. This approach is effective in various environments.

[0004] Existing deep learning-based infrared and visible light target detection methods mainly involve equal processing of multimodal features. These methods extract features from multimodal features using a non-shared weight encoder and then input both types of features into a fusion module to complete the fusion, or perform feature fusion during the decoding process. Existing infrared and visible light target detection methods have the following problems: (1) They do not dynamically measure the feature effects of infrared and visible light modes, making it impossible for the fused features to guarantee sufficient and accurate representation of salient targets. (2) The differences in modal information between infrared and visible light modes are not fully reflected during feature extraction. The prominent parts of the two modal features are different, resulting in low quality of fused feature extraction. (3) Due to the differences in the prominent parts of the multimodal features themselves, existing methods cannot guarantee the actual effect of multimodal feature interaction fusion, and the fused features cannot guarantee full utilization of infrared features. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems existing in infrared and visible light target detection technologies, such as the lack of dynamic measurement of the characteristics of infrared and visible light modes, the failure to fully reflect the differences in modal information between infrared and visible light modes, and the inability to fully utilize infrared features after fusion. This invention provides a salient target detection method based on infrared and visible light image fusion, the method comprising:

[0006] S1: Input corresponding infrared and visible light image pairs as the dataset, and preprocess the training set images in the dataset to obtain the preprocessed training set;

[0007] S2: Construct a multimodal salient object detection network, which includes a multimodal effect prediction module, an encoder, a multimodal feature group fusion module, a multimodal refinement fusion module CRM, a decoder, and convolutional blocks;

[0008] The multimodal salient target detection network obtains the feature weights of visible light based on the multimodal effect prediction module. The multimodal feature group fusion module fuses the features extracted by the encoder based on the weights and outputs the results to the multimodal refinement fusion module for refinement. The refined features are input into the decoder for decoding and output as a salient target detection mask through a convolutional block.

[0009] S3: Set the loss function, and train the multimodal salient object detection network according to the loss function and the preprocessed training set to obtain the multimodal salient object detection model;

[0010] S4: Input any infrared and visible light image pair into the multimodal salient target detection model to obtain salient target detection results.

[0011] Preferably, the preprocessing in step S1 includes: flipping the input image horizontally, rotating it, randomly cropping it, and enhancing its color, and then standardizing the image pixel values ​​to a given mean and standard deviation.

[0012] Preferably, the specific process of the multimodal saliency target detection network in step S2 is as follows:

[0013] S21: Input the visible light image to the multimodal action prediction module to obtain the dynamic weight values ​​of the visible light features; input the visible light image and its corresponding infrared image to the visible light image encoder and the infrared image encoder to extract infrared features and visible light features at different scales;

[0014] S22: Input infrared features and visible light features of the same scale into a multimodal feature group fusion module to obtain the preliminary fused features at that scale;

[0015] S23: Input the preliminary fusion features and infrared features of the same scale into a multimodal refinement fusion module to obtain the refined fusion features of that scale;

[0016] S24: The refined fusion features at the same scale are input into the decoder corresponding to that scale. The decoder outputs the result to the next level decoder, and the output of the highest level decoder is used as the final decoding result.

[0017] S25: The decoding result is input into the convolutional block to obtain a salient target detection mask.

[0018] Preferably, the multimodal effect prediction module process includes: based on prior perception of illumination, calculating the perceived brightness of all pixels in the input visible light image, and calculating the average perceived brightness of all pixels to obtain the average perceived brightness. The average perceived brightness is then processed by a neural network to obtain the dynamic weights of the visible light features. The specific process can be formulated as follows:

[0019]

[0020] Among them, w rgb Let B(x) be the dynamic weight of the visible light features, B(x) be the fully connected operation and activation function of the neural network, n be the number of image pixels, and w1, w2, and w3 be the weights of the illumination estimation formula, which are set to 0.299, 0.587, and 0.114 respectively in the formula, corresponding to I... R I G I B These are the values ​​of the R, G, and B channels of the visible light image, respectively, and x and y are the specific horizontal and vertical coordinates in the image.

[0021] Preferably, the encoder in step S21 is an asymmetric ResNet50-ResNet34 encoder structure, wherein the visible light image encoder is a ResNet50 with the output layer removed, and the infrared image encoder is a ResNet34 with the output layer removed.

[0022] Preferably, step S22 specifically includes:

[0023] S221: Perform spatial attention calculation on visible light features at the same scale, and adjust the infrared features according to the spatial attention calculation results and the dynamic weights of the visible light features to obtain the infrared features adjusted for that scale.

[0024] S222: The infrared and visible light features adjusted to the same scale are initially fused using a convolutional neural network to obtain the initial fused features at that scale.

[0025] Preferably, step S222 includes: dividing the feature dimensions of visible light features and infrared features at the same location at the same scale into a corresponding number of groups; performing weighted fusion of visible light features and infrared features within each group according to the dynamic weight of the visible light features; then using convolutional blocks within a single group to perform intra-group information fusion of the weighted fused features; and then using convolutional blocks with the same kernel size between different groups to perform inter-group information fusion of the intra-group information fused features, thereby obtaining the preliminary fused features at that scale.

[0026] Preferably, step S23 specifically includes:

[0027] S231: Map the initial fusion features and infrared features of the same scale to block vectors of the corresponding number of fusion features and block vectors of the corresponding number of infrared features;

[0028] S232: After the block vector is normalized by the layer, the correlation vector of the query vector corresponding to the block vector of the fused feature and the key vector corresponding to the block vector of the infrared feature are calculated to obtain the correlation vector of the block vector of the fused feature and the block vector of the infrared feature at this scale.

[0029] S233: Adjust the dimensions of the correlation vector to obtain the importance of the block vector of the infrared features at this scale to the fused features;

[0030] S234: Based on the dynamic weight values ​​of the importance features and visible light features, obtain the enhanced fusion features at this scale;

[0031] S235: Perform residual connection on the result of convolution of the enhanced fusion feature and the enhanced fusion feature, and then perform batch normalization to obtain the fusion feature refined at this scale.

[0032] Preferably, step S234 includes: performing convolution operations on importance features at the same scale, then using bilinear interpolation to upsample the importance features after convolution, then performing convolution operations on the sampled importance features, then obtaining the weight value required to enhance the fusion feature based on the operation result and the dynamic weight value of visible light, finally multiplying the infrared feature with the weight value and adding it to the fusion feature, and then batch normalizing the added feature to obtain the enhanced fusion feature at that scale.

[0033] Preferably, the multimodal saliency target detection network is trained using a loss function composed of cross-entropy loss and IOU loss function. The expression for the total loss function is as follows:

[0034] L = L bce (s,μ)+L iou (s,μ)

[0035] Among them, L bce (s,μ) represents the cross-entropy loss function, L iou (s,μ) represents the IOU loss function of the network, where s is the detection result obtained by the multimodal saliency object detection model and μ is the true label provided by the training set.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0037] 1. This invention proposes a multimodal action prediction module that uses a neural network to evaluate image illumination and obtain dynamic weights of visible light features. It can dynamically adjust the feature ratio in the subsequent detection of fused features according to illumination conditions. It makes full use of visible light images when the illumination is good, while relying more on infrared images when the illumination is insufficient. This can effectively improve the robustness of fused features to various complex environments.

[0038] 2. This invention proposes a multimodal feature group fusion module. After adjusting the infrared features, the multimodal features are grouped, fused by group, and then fused as a whole. This completes the fusion of modal features from space to channel and from intra-group to global features, and can obtain high-quality fused features with low computational and parameter requirements.

[0039] 3. This invention proposes a multimodal refinement fusion module, which, based on the initial feature fusion, performs in-depth mining of the relationship between the fused features and the additional modal features, avoiding insufficient fusion of the additional modal features, and completing the refinement and enhancement of the fused features, giving them better scene representation capabilities and improving the performance of salient target detection. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the overall process of the target detection method according to an embodiment of the present invention.

[0041] Figure 2 This is a diagram illustrating the overall framework of the multimodal saliency target detection network according to an embodiment of the present invention.

[0042] Figure 3 This is a schematic diagram of the multimodal action prediction module according to an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the multimodal feature group fusion module according to an embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the multimodal refinement and fusion module according to an embodiment of the present invention;

[0045] Figure 6 This is a comparison chart of the detection performance between the target detection method of the present invention and existing salient target detection methods;

[0046] Figure 7This is a comparison chart of the salient target detection results of the target detection method of the present invention under low light conditions with the real labels of the dataset. Detailed Implementation

[0047] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0048] Example 1

[0049] like Figure 1 As shown, this invention provides a salient target detection method based on infrared and visible light image fusion, the method comprising:

[0050] S1: Input corresponding infrared and visible light image pairs as the dataset, and preprocess the training set images in the dataset to obtain the preprocessed training set;

[0051] Specifically, the datasets used in this invention are the publicly available infrared and visible light salient target detection datasets VT5000 and VT821, which include infrared and visible light image pairs under various environments and real label images. Half of the VT5000 dataset is used as the training set, and the VT821 dataset is used as the test set. The images in the training set are subjected to horizontal flipping and rotation operations with a probability of 0.5, followed by random cropping and color enhancement. Finally, the image pixel values ​​are standardized to a given mean and standard deviation, where the mean is (0.485, 0.456, 0.406) and the standard deviation is (0.229, 0.224, 0.225), thus completing the preprocessing of the training set.

[0052] S2: Construct a multimodal salient object detection network, which includes a multimodal effect prediction module, an encoder, a multimodal feature group fusion module, a multimodal refinement fusion module CRM, a decoder, and convolutional blocks;

[0053] Specifically, the overall framework of the multimodal saliency target detection network constructed in this invention is as follows: Figure 2 As shown, the specific network flow is as follows:

[0054] S21: Input the visible light image to the multimodal action prediction module to obtain the dynamic weight values ​​of the visible light features; input the visible light image and its corresponding infrared image to the encoder to extract infrared features and visible light features at different scales;

[0055] Specifically, the multimodal action prediction module proposed in this invention is as follows: Figure 3As shown, this module, based on prior perception of illumination, uses the Luminance illumination estimation method, a traditional brightness estimation method, to predict modal effects. The Luminance illumination estimation formula calculates the weighted sum of squares of the visible light images to determine the perceived brightness. The estimate is calculated using the following formula:

[0056]

[0057] Where w1, w2, and w3 are the weights in the illumination estimation formula, traditional brightness estimation methods widely use a weighted sum calculation method based on ITU-R BT.601 to calculate the grayscale of each pixel in a visible light image, ultimately obtaining a grayscale map of the visible light image. The grayscale value of each pixel is then used as an engineering approximation of the true brightness. Based on this, existing research on the sensitivity of the human eye to three colors sets the three weights w1, w2, and w3 to 0.299, 0.587, and 0.114 respectively, indicating that the human eye's sensitivity to color is, in descending order, green, red, and blue. The corresponding I... R I G I B These are the values ​​of the R, G, and B channels of the visible light image, respectively, and x and y are the specific horizontal and vertical coordinates in the image.

[0058] Furthermore, the multimodal effect prediction module uses the Luminance illumination estimation formula to calculate the perceived brightness of the visible light image. After obtaining the perceived brightness of all pixels in the visible light image, it calculates the average perceived brightness of all pixels to obtain the average perceived brightness of the current visible light image for the captured scene. The average perceived brightness is input into the fully connected layer of the neural network B(x) to increase the perceived brightness dimension and remap it to the weight domain. The weight range is then normalized using the Sigmoid activation function. Since removing the square root from the Luminance estimation formula keeps the overall variation region unchanged and significantly reduces computation, the specific calculation process of this module can be described as follows:

[0059]

[0060] Among them, w rgb Let B(x) be the dynamic weight of the visible light features, B(x) be the fully connected operation and activation function of the neural network, n be the number of image pixels, and w1, w2, and w3 be the weights of the illumination estimation formula, which are set to 0.299, 0.587, and 0.114 respectively in the formula, corresponding to I... R I G I B These are the values ​​of the R, G, and B channels of the visible light image, respectively, and x and y are the specific horizontal and vertical coordinates in the image.

[0061] Furthermore, since infrared and visible light features contain different amounts of information, using the same encoder or a symmetric encoder cannot represent the characteristics of multimodal features. Therefore, an asymmetric encoder structure is chosen for multimodal, multi-level feature extraction. This invention is based on an asymmetric ResNet50-ResNet34 encoder structure, where the visible light image encoder is a ResNet50 with the output layer removed, and the infrared image encoder is a ResNet34 with the output layer removed. After the visible light image and infrared feature image are input into the corresponding encoder, they pass through the input layer, and multiple residual blocks perform multi-scale feature extraction to extract visible light features at different scales. With infrared features

[0062] S22: Input infrared features and visible light features of the same scale into a multimodal feature group fusion module to obtain the preliminary fused features at that scale;

[0063] Specifically, the multimodal feature group fusion module proposed in this invention is as follows: Figure 4 As shown, this module performs preliminary fusion of the extracted infrared and visible light features. The specific process is as follows:

[0064] S221: Perform spatial attention calculation on visible light features at the same scale, and adjust the infrared features according to the spatial attention calculation results and the dynamic weights of the visible light features to obtain the infrared features adjusted for that scale.

[0065] Specifically, since the salient parts of infrared and visible light features may not be consistent, it is necessary to induce infrared features to focus more on visually salient regions. Therefore, this module uses the spatial attention calculation results of visible light features to adjust the infrared features. The specific implementation of spatial attention calculation is as follows: the visible light features are concatenated according to the maximum value and average value of each channel to obtain a two-channel feature map. Then, a convolutional block with a kernel size of 3, a stride of 1, and padding of 1 is used to learn the spatial importance of the feature map to obtain the spatial attention calculation result. Finally, based on the spatial attention calculation result and the dynamic weights of the visible light features, the infrared feature information at the corresponding position is extracted and added to the original infrared features, thereby amplifying the value of the visually salient regions in the infrared features and completing the adjustment of the infrared features. This process can be formulated as follows:

[0066]

[0067] in, The adjusted infrared features of the i-th layer, For the infrared features of the i-th layer before adjustment, w rgb The dynamic weights for visible light features. and These represent the average and maximum values ​​of the visible light features by channel, respectively. Sigmoid(x) is the normalization function, and conv(x) is the convolution function.

[0068] S222: The infrared and visible light features adjusted to the same scale are initially fused using a convolutional neural network to obtain the initial fused features at that scale.

[0069] Specifically, after adjusting the infrared features, the infrared and visible light features are fused. This involves fusing visible light features at the same location on the same scale. With adjusted infrared features The feature dimensions are divided into N i Groups, N i For the first to fourth scale features, the values ​​are [1, 2, 4, 8]. Based on the dynamic weights of the visible light features, the visible light features and the adjusted infrared features within each group are weighted and fused. Then, within a single group, a kernel of 3, a stride of 1, a padding of 1, and a grouping of N are used. i The convolutional blocks perform intra-group information fusion on the weighted fused features. Then, convolutional blocks with the same kernel size are used between different groups to perform inter-group information fusion on the intra-group fused features, thus obtaining the preliminary fused features at this scale. The calculation formula is as follows:

[0070]

[0071] in, The fused features are those of the i-th layer. The visible light characteristics of the i-th layer, For the adjusted infrared features of the i-th layer, w rgb The dynamic weights for visible light features. Grouped into N i The convolution operation, N i The values ​​are [1,2,4,8], and conv(x) represents the convolution operation between different groups.

[0072] S23: Input the preliminary fusion features and infrared features of the same scale into a multimodal refinement fusion module to obtain the refined fusion features of that scale;

[0073] Specifically, the multimodal refinement and fusion module proposed in this invention is as follows: Figure 5 As shown, this module uses correlation calculation to refine and enhance the initially fused features, avoiding insufficient fusion of additional modal features, thus giving the fused features better scene representation capabilities and improving the performance of salient target detection. The specific process of the multimodal refinement fusion module is as follows:

[0074] S231: Map the initial fusion features and infrared features of the same scale to block vectors of the corresponding number of fusion features and block vectors of the corresponding number of infrared features;

[0075] Specifically, the initial fusion features will be... and infrared features Convolution operations are performed using convolutional blocks with a kernel size of 3, a stride of 3, and padding of 0 to obtain block vectors of the corresponding number of fused features. And the block vector of the corresponding number of infrared features Where B is the number of infrared and visible light image pairs, C i N represents the channel length of the corresponding block vector. i N represents the number of block vectors at the i-th scale. i The values ​​for the first-scale features to the fourth-scale features are [8, 4, 2, 1].

[0076] S232: After the block vector is normalized by the layer, the correlation vector of the query vector corresponding to the block vector of the fused feature and the key vector corresponding to the block vector of the infrared feature are calculated to obtain the correlation vector of the block vector of the fused feature and the block vector of the infrared feature at this scale.

[0077] Specifically, the correlation calculation formula for fused features and infrared features at the same scale is as follows:

[0078]

[0079] in, For the correlation vector, The block vector of the fused features at the i-th scale is linearly mapped through a fully connected layer to generate the query vector. The block vector of infrared features at scale i is linearly mapped through a fully connected layer to generate a key vector, d. k Let x be the vector dimension, and Softmax(x) be the normalization function.

[0080] S233: Adjust the dimensions of the correlation vector to obtain the importance of the block vector of the infrared features at this scale to the fused features;

[0081] Specifically, the dimension of the correlation vector is B×1×N. i The importance feature r is obtained by adjusting its dimensions. i r i The dimension is B×1×H×W, where the height H and width W of the feature are both B×1×H×W.

[0082] S234: Based on the dynamic weight values ​​of the importance features and visible light features, obtain the enhanced fusion features at this scale;

[0083] Specifically, after convolving the importance features with a kernel size of 3, stride of 1, and padding of 1, bilinear interpolation is used to upsample the convolved importance features. Finally, the same convolving block is used to convolve the sampled importance features again. The weight values ​​required to enhance the fusion features are obtained based on the calculation result and the dynamic weight values ​​of visible light. The infrared features are then multiplied by these weight values ​​and added to the fusion features. Batch normalization is then applied to the added features to obtain the enhanced fusion features. The enhancement process of the initial fusion features at four scales can be formulated as follows:

[0084]

[0085] in, For the enhanced fusion features at the i-th scale, For the initial fusion features at the i-th scale, r i Let represent the importance of the block vectors of infrared features at the i-th scale compared to the fused features. Let x be the infrared feature at the i-th scale, conv(x) be the convolution function, up(x) be the bilinear interpolation upsampling operation, and BN(x) be the batch normalization.

[0086] S235: Perform residual connection on the result of convolution of the enhanced fusion feature and the enhanced fusion feature, and then perform batch normalization to obtain the fusion feature refined at this scale.

[0087] Specifically, the enhanced fusion features are convolved using a convolutional block with a kernel of 3, a stride of 1, and padding of 1, and residual connections are made with the enhanced fusion features to ensure that no information of the enhanced fusion features is lost. After batch normalization, the refined fusion features are obtained. The process can be formalized as follows:

[0088]

[0089] in, Let conv(x) be the enhanced fusion feature at the i-th scale, conv(x) be the convolution function, and BN(x) be the batch normalization function.

[0090] S24: The refined fusion features at the same scale are input into the decoder corresponding to that scale. The decoder outputs the result to the next level decoder, and the output of the highest level decoder is used as the final decoding result.

[0091] Specifically, each refined fused feature corresponds to a different level of decoder according to scale from low to high. The lowest level decoder directly inputs the refined fused feature of the corresponding scale. The inputs of the other decoders are the refined fused feature of the corresponding scale and the output of the next lower level decoder. The output of the highest level decoder is used as the final decoding result. In each decoder layer, all inputs are summed, and the features are processed by a convolutional block with a kernel size of 3, a stride of 1, and padding of 1. Bilinear interpolation is then used to double the feature resolution to complete the feature decoding.

[0092] S25: The decoding result is used to perform convolution operation on a 1x1 convolution block to obtain a salient target detection mask.

[0093] S3: Set the loss function, and train the multimodal salient object detection network according to the loss function and the preprocessed training set to obtain the multimodal salient object detection model;

[0094] Specifically, the salient object detection network of this invention is implemented using PyTorch. The network is trained using the AdamW optimizer and a loss function composed of cross-entropy loss and IOU loss. The optimizer parameters are set as follows: the initial learning rate is Ie. -4 The momentum is 0.9, the decay rate per generation is 0.999, and 400 training rounds are performed with a batch size of 8. The parameters of the multimodal saliency target detection model are optimized using a stochastic gradient descent and backpropagation algorithm to address the network's total loss. The training total loss function L is as follows:

[0095] L = L bce (s,μ)+L iou (s,μ)

[0096] Among them, L bce (s,μ) represents the cross-entropy loss function, L iou (s,μ) represents the IOU loss function of the network, where s is the detection result obtained by the multimodal saliency object detection model and μ is the true label provided by the training set.

[0097] Furthermore, the formula for the cross-entropy loss function is as follows:

[0098] L bce (s,μ)=-(μlogs+(1-μ)log(1-s))

[0099] Furthermore, the formula for the IOU loss function is as follows:

[0100]

[0101] S4: Input any infrared and visible light image pair into the multimodal salient target detection model to obtain salient target detection results.

[0102] Specifically, the publicly available dataset VT821 is used as the test set to input the trained multimodal salient object detection model, and salient object detection results are obtained. A comparison of the detection performance of the object detection method of this invention with existing salient object detection methods is provided below. Figure 6 As shown in the figure, the detection results of this method have more correct detections and fewer false positives at edges. The detected salient target structures and details are more similar to the real labels. A comparison of the salient target detection results of this invention under low-light conditions with the real labels in the dataset is also provided. Figure 7 As shown in the figure, under extremely low light conditions, the detection results of the detection method of the present invention are less affected and have high robustness to environmental impact.

[0103] Furthermore, the salient target detection method of this invention can obtain high-quality fused features with lower computational cost and parameter count. The table below compares the floating-point operations per second (Flops) and parameter count required by this method with those of the MISD, CGFNet, and CAVER methods:

[0104] Flops(G) Param(M) MISD 434.43 52.43 CGFNet 750.56 66.382 CAVER 44.44 55.80 This method 41.69 51.14

[0105] Furthermore, the comparison results of the detection metrics of the salient target detection method of the present invention with those of the ECFFNet method, MISD method, CGFNet method, and CAVER method are shown in the following table:

[0106]

[0107] In the table, MAE (Mean Absolute Error) represents the average absolute pixel error. For a given resolution, the smaller the value of MAE, the smaller the global absolute pixel error. F m Frecall represents the ratio of precision to recall in the model's detection of target regions. It is used to evaluate how well the model correctly distinguishes target regions; a higher Frecall indicates a stronger ability to correctly distinguish them. m The non-negative weights for recall and precision are set to 0.3; S m (Structure measure) represents a structural measurement index. It judges the overall structural similarity by focusing on the structural similarity between the saliency map and the ground truth labels provided by the dataset, considering both region-perceived and object-perceived structural similarity. The higher the index, the more similar the predicted region is to the real region in structure; E m(Enhanced-alignment measure) represents the enhanced alignment measurement metric, a metric capable of performing image-level and local pixel matching statistics; a higher metric indicates a better match. The table shows the metrics MAE and S of this method. m and F m All are optimal, indicator F m Second only to the CAVER method.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A salient target detection method based on infrared and visible light image fusion, characterized in that, The method includes the following steps: S1: Input corresponding infrared and visible light image pairs as the dataset, and preprocess the training set images in the dataset to obtain the preprocessed training set; S2: Construct a multimodal salient object detection network, which includes a multimodal effect prediction module, an encoder, a multimodal feature group fusion module, a multimodal refinement fusion module CRM, a decoder, and convolutional blocks; The multimodal salient target detection network obtains the feature weights of visible light based on the multimodal effect prediction module. The multimodal feature group fusion module fuses the features extracted by the encoder based on the weights and outputs the results to the multimodal refinement fusion module for refinement. The refined features are input into the decoder for decoding and output as a salient target detection mask through a convolutional block. S3: Set the loss function, and train the multimodal salient object detection network according to the loss function and the preprocessed training set to obtain the multimodal salient object detection model; S4: Input any infrared and visible light image pair into the multimodal salient target detection model to obtain the salient target detection result; The specific process of the multimodal saliency target detection network is as follows: S21: Input the visible light image to the multimodal action prediction module to obtain the dynamic weight values ​​of the visible light features; input the visible light image and its corresponding infrared image to the visible light image encoder and the infrared image encoder to extract infrared features and visible light features at different scales; S22: Input infrared features and visible light features of the same scale into a multimodal feature group fusion module to obtain the preliminary fused features at that scale; S23: Input the preliminary fusion features and infrared features of the same scale into a multimodal refinement fusion module to obtain the refined fusion features of that scale; S24: The refined fusion features at the same scale are input into the decoder corresponding to that scale. The decoder outputs the result to the next level decoder, and the output of the highest level decoder is used as the final decoding result. S25: The decoding result is input into the convolutional block to obtain a salient target detection mask; The specific steps in S23 are as follows: S231: Map the initial fusion features and infrared features of the same scale to block vectors of the corresponding number of fusion features and block vectors of the corresponding number of infrared features; S232: After the block vector is normalized by the layer, the correlation vector of the query vector corresponding to the block vector of the fused feature and the key vector corresponding to the block vector of the infrared feature are calculated to obtain the correlation vector of the block vector of the fused feature and the block vector of the infrared feature at this scale. S233: Adjust the dimensions of the correlation vector to obtain the importance of the block vector of the infrared features at this scale to the fused features; S234: Based on the dynamic weight values ​​of the importance features and visible light features, obtain the enhanced fusion features at this scale; S235: Perform residual connection on the result of convolution of the enhanced fusion feature and the enhanced fusion feature, and then perform batch normalization to obtain the fusion feature refined at this scale; S234 includes: performing convolution operations on importance features at the same scale, then using bilinear interpolation to upsample the importance features after convolution, then performing convolution operations on the sampled importance features, then obtaining the weight value required to enhance the fusion feature based on the operation result and the dynamic weight value of visible light, finally multiplying the infrared feature with the weight value and adding it to the fusion feature, and then batch normalizing the added feature to obtain the enhanced fusion feature at that scale.

2. The salient target detection method based on infrared and visible light image fusion as described in claim 1, characterized in that, The preprocessing includes: flipping the input image horizontally, rotating it, randomly cropping it, and enhancing its color, and then standardizing the image pixel values ​​to a given mean and standard deviation.

3. The salient target detection method based on infrared and visible light image fusion as described in claim 1, characterized in that, The specific process of the multimodal effect prediction module includes: based on prior perception of illumination, calculating the perceived brightness of all pixels in the input visible light image, and calculating the average perceived brightness of all pixels to obtain the average perceived brightness. The average perceived brightness is then processed by a neural network to obtain the dynamic weights of the visible light features. The specific process can be formulated as follows: in, w rgb The dynamic weights for visible light features. For fully connected operations and activation functions in a neural network, The number of pixels in the image. , , The weights in the illumination estimation formula are set to 0.299, 0.587, and 0.114 respectively. , , These represent the values ​​of the R, G, and B channels of a visible light image. , These are the specific horizontal and vertical coordinates in the image.

4. The salient target detection method based on infrared and visible light image fusion as described in claim 1, characterized in that, The encoder is an asymmetric ResNet50-ResNet34 encoder structure, wherein the visible light image encoder is a ResNet50 with the output layer removed, and the infrared image encoder is a ResNet34 with the output layer removed.

5. The salient target detection method based on infrared and visible light image fusion as described in claim 1, characterized in that, The specific steps of S22 are as follows: S221: Perform spatial attention calculation on visible light features at the same scale, and adjust the infrared features according to the spatial attention calculation results and the dynamic weights of the visible light features to obtain the infrared features adjusted for that scale. S222: The infrared and visible light features adjusted to the same scale are initially fused using a convolutional neural network to obtain the initial fused features at that scale.

6. The salient target detection method based on infrared and visible light image fusion as described in claim 5, characterized in that, S222 includes: dividing the feature dimensions of visible light features and infrared features at the same location at the same scale into a corresponding number of groups; performing weighted fusion of visible light features and infrared features within each group according to the dynamic weight of visible light features; then using convolutional blocks to perform intra-group information fusion on the weighted fused features within a single group; and then using convolutional blocks with the same kernel size to perform inter-group information fusion on the intra-group information fused features between different groups, thereby obtaining the preliminary fused features at that scale.

7. A salient target detection method based on infrared and visible light image fusion as described in any one of claims 1-6, characterized in that, The multimodal salient object detection network is trained using a loss function comprised of both cross-entropy and IOU loss functions. The expression for the total loss function is as follows: in, Represents the cross-entropy loss function. This represents the IOU loss function of the network. The detection results are obtained from the multimodal saliency target detection model. The real labels provided for the training set.

Images