Infrared visible light image fusion method for complex lighting environment

Abstract

The invention provides an infrared and visible light image fusion method for a complex lighting environment, and the method comprises the steps: obtaining a plurality of pairs of infrared images and visible light images after registration in a plurality of scenes to form a training image subset, thereby obtaining a training data set, and carrying out the identification of the training data set through employing a scene perception network; inputting the training image subset identified as the night scene into a mask modeling algorithm to generate a mask for guiding network training; constructing a layered infrared visible light image fusion network; and constructing a loss function to guide training of the layered infrared visible light image fusion network. According to the invention, fusion loss is constructed by using a scene sensing network and a mask modeling algorithm so as to adapt to different illumination scenes; a gradient enhancement module is designed to enhance texture information in a mixed mode on a feature level; according to the method, a multi-scale semantic interaction module is introduced into a deep semantic level, so that semantic information in bimodal features is effectively utilized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an infrared and visible light image fusion method in a complex lighting environment. Background Art

[0002] At present, images taken by single-modal sensors cannot effectively and comprehensively describe the imaging scene; visible light sensors can provide background details with high spatial resolution through reflected light imaging, but they cannot clearly see the target when the lighting or camouflage conditions are poor; in contrast, infrared sensors use the differentiated thermal radiation emitted by objects for imaging, are not affected by harsh conditions, and can work around the clock. This complementary feature has led to the emergence of infrared-visible light image fusion, and the fused image can highlight the target and reflect rich texture details. Therefore, infrared and visible light image fusion has been widely used as a preprocessing module for advanced vision tasks such as target detection, pedestrian re-identification, and semantic segmentation.

[0003] Existing methods are often designed for conditions with uniform illumination distribution. Infrared and visible light image fusion may face a variety of scenarios, including day and night. In daytime scenes, the light source is evenly distributed, and visible light images contain more meaningful information. In night scenes, the light source distribution is extremely uneven, and there is a common situation where the brightness of the light area is too high, that is, local exposure. When faced with point local exposure, existing methods cannot reasonably utilize the detailed information of the local area, resulting in serious loss of details and unable to generate a fused image that conforms to human vision and is beneficial to downstream advanced vision tasks. Summary of the invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defects mentioned above, thereby providing an infrared and visible light image fusion method for complex lighting environments.

[0005] In order to solve the above problems, the present invention provides an infrared and visible light image fusion method for complex lighting environments, which comprises the following steps:

[0006] S1: Acquire multiple pairs of infrared images and visible light images after registration in various scenes to form training image subsets to obtain training data sets, and identify the training data sets using a scene perception network;

[0007] S2: Input a subset of training images identified as nighttime scenes into a mask modeling algorithm to generate masks to guide network training;

[0008] S3: Constructing a hierarchical infrared and visible light image fusion network;

[0009] S4: Construct a loss function to guide the training of the hierarchical infrared-visible image fusion network.

[0010] Preferably, in S1, the scene perception network includes: a convolutional layer, a batch normalization layer, a ReLu activation function, a maximum pooling layer and a fully connected layer;

[0011] The training data set is identified by using a scene perception network as follows: outputting the probability that the training data set belongs to a daytime scene and a nighttime scene.

[0012] Preferably, in S2, the mask modeling algorithm is: inputting the visible light image in the training image subset into the scene perception network for identification, performing Gaussian filtering on the image identified as a night scene, binarizing the result of the Gaussian filtering, processing the binarized result, and then superimposing the processed binarized result with the character semantic segmentation label to generate a mask to guide network training.

[0013] Preferably, the hierarchical fusion network includes: a shallow feature fusion layer, a middle gradient enhancement layer and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module FMEM for enhanced extraction of shallow features, the middle gradient enhancement layer includes a gradient enhancement module GEM for gradient enhancement of middle mixed features, and the deep semantic interaction layer includes a multi-scale semantic interaction module MSIM for integrating deep features.

[0014] Preferably, the hierarchical infrared-visible light image fusion network is constructed in S3 as follows:

[0015] Input the infrared image and visible light image of the training data set into the encoder of the network model to extract features, extract the features of the i-th layer respectively, and then input the features of the infrared image and visible light image in the shallow layer into the feature reuse module after being concatenated in the channel dimension for further feature extraction and reuse;

[0016] A gradient enhancement module is designed in the middle layer to enhance the gradient texture features at the mixed feature level, and to connect the infrared image and visible light image features in series in the channel dimension.

[0017] In the deep layer, a multi-scale semantic interaction module based on multi-scale cross-attention is designed to integrate deep features;

[0018] The features of each layer are input into the decoder and connected in series with the features of the previous layer in the channel dimension to restore the fused image.

[0019] Preferably, the gradient enhancement module is designed in the middle layer as follows:

[0020] The mixed features of the input gradient enhancement module are first passed through a layer of convolution, and then the Sobel operator, Laplacian operator and Prewitt operator are used to calculate the gradient of the features output by the first layer of convolution respectively. The gradient features calculated by the three gradient operators are then connected in series in the channel dimension and sent to the convolution layer, and then connected in series with the mixed features of the initial input in the channel dimension and sent to the convolution layer.

[0021] Preferably, the multi-scale semantic interaction module is:

[0022] Three convolutional layers with different kernel sizes are used to capture multi-scale deep features. The features at different scales are then concatenated in the channel dimension. A projection function containing a convolution kernel reshaping operation is deployed to convert the multi-scale features into keys and values. The modal attention map corresponding to each modality x is calculated, and the value is multiplied by the corresponding attention to obtain features with global context. The global features are then reshaped, and a 1×1 convolution is used to adjust the channel and then connected in the channel dimension to generate a fused feature rich in semantics.

[0023] Preferably, the constructed loss function guides the training of the hierarchical infrared-visible light image fusion network by utilizing a weighted combination of a global fusion loss and a local information supplementary fusion loss to obtain a total loss of the fusion network.

[0024] Preferably, the global fusion loss is: intensity loss, texture loss, style loss;

[0025] Among them, for daytime conditions, the light source is single, and using the intensity loss with the maximum pixel value is reasonable for extracting salient areas from the two modalities;

[0026] For nighttime situations, due to the existence of local area exposure, the mask generated by the mask modeling algorithm is used to construct the intensity loss at night. The scene probability output by the SAN is used to reasonably guide the network to avoid the local light source exposure area in the visible light image and retain the salient targets and textures in the area from the infrared modality.

[0027] Preferably, the local information supplementary fusion loss is:

[0028] Considering the serious information loss in the local exposure area of ​​the visible light image, the local information supplement fusion loss is used to guide the network to integrate the information from the infrared modality into the fused image.

[0029] The infrared and visible light image fusion method provided by the present invention for complex lighting environments has the following beneficial effects:

[0030] The present invention constructs a fusion loss using a scene perception network and a mask modeling algorithm to adapt to different lighting scenes; it also enhances the texture information in the mixed modality at the feature level by designing a gradient enhancement module; the present invention also effectively utilizes the semantic information in the bimodal features by introducing a multi-scale semantic interaction module at the deep semantic level. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the framework of the infrared and visible light image fusion method of the present invention;

[0032] Figure 2 It is a schematic diagram of the structure of the gradient enhancement module of the present invention;

[0033] Figure 3 It is a schematic diagram of the structure of the multi-scale semantic interaction module of the present invention;

[0034] Figure 4 It is a schematic diagram of the comparison results of the simulation experiment of the present invention;

[0035] Figure 5 This is a schematic diagram of the complete results of semantic segmentation using DeepLabV3+ in the simulation experiment of the present invention. DETAILED DESCRIPTION

[0036] like Figure 1-5 As shown, the present invention provides an infrared and visible light image fusion method for a complex lighting environment, which comprises the following steps:

[0037] S1: Acquire multiple pairs of infrared images and visible light images after registration in various scenes to form training image subsets to obtain training data sets, and identify the training data sets using a scene perception network;

[0038] S2: Input a subset of training images identified as nighttime scenes into a mask modeling algorithm to generate masks to guide network training;

[0039] S3: Constructing a hierarchical infrared and visible light image fusion network;

[0040] S4: Construct a loss function to guide the training of the hierarchical infrared-visible image fusion network.

[0041] In some embodiments, in S1, the scene perception network includes: a convolutional layer, a batch normalization layer, a ReLu activation function, a maximum pooling layer, and a fully connected layer;

[0042] The training data set is identified by using a scene perception network as follows: outputting the probability that the training data set belongs to a daytime scene and a nighttime scene.

[0043] In some embodiments, in S2, the mask modeling algorithm is: input the visible light image in the training image subset into the scene perception network for identification, perform Gaussian filtering on the image identified as a night scene, set the area with a pixel value greater than 300 and a cumulative pixel point greater than 70 to 1, and the rest to 0, and binarize the result of Gaussian filtering, set the area with a pixel value greater than 300 and a cumulative pixel point greater than 70 to 1, and the rest to 0, and then process the binarization result, and then superimpose the processed binarization result with the character semantic segmentation label to generate a mask M to guide network training.

[0044] Specifically, the processing includes: performing morphological corrosion and dilation processing on the binarization result to eliminate noise.

[0045] In some embodiments, the hierarchical fusion network includes: a shallow feature fusion layer, a middle gradient enhancement layer and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module FMEM for enhanced extraction of shallow features, the middle gradient enhancement layer includes a gradient enhancement module GEM for gradient enhancement of middle mixed features, and the deep semantic interaction layer includes a multi-scale semantic interaction module MSIM for integrating deep features.

[0046] In some embodiments, the hierarchical infrared-visible light image fusion network constructed in S3 is:

[0047] The infrared image I of the training dataset ir ∈R H×W×1 and visible light image I vi ∈R H×W×3 Input the encoder of the network model to extract features. H×W×1 represents an infrared image with a height of H, a width of W, and a channel number of 1. H×W×3 represents a visible light image with a height of H, a width of W, and a channel number of 3. The formula is:

[0048] {F ir , F vi}={E(I ir ), E(I vi )}

[0049] where F ir and F vi represents infrared features and visible light features, E(·) represents the encoder;

[0050] Extract the features of layer i respectively, and They are defined as the features extracted from infrared images and visible light images by the i-th layer of the network, respectively, and the formula is:

[0051]

[0052] in represents the infrared and visible light features extracted by the network at layer i, and represents the infrared feature extraction block and visible light feature extraction block of the i-th layer of the encoder, where i=1, 2, 3, 4, 5. When i=1, and The input is a visible light image and a visible light image. When i=2, 3, 4, 5, and The input is the features of the two modes of the previous layer;

[0053] The features of the shallow infrared image and visible light image are connected in series in the channel dimension and input into the feature reuse module for further feature extraction and reuse. The formula is:

[0054]

[0055] represents the concatenated features, concat(·) represents the concatenation in the channel dimension, fu1 represents the fused features obtained in the first layer of the network, and FMEM{·} represents the feature reuse extraction module;

[0056] A gradient enhancement module is designed in the middle layer to enhance the gradient texture features at the mixed feature level, and the infrared image and visible light image features are connected in series in the channel dimension. The formula is:

[0057]

[0058] Where i = (2, 3), indicating the features of the i-th layer, Represents the result after concatenation in the channel dimension;

[0059] In the deep layer, a multi-scale semantic interaction module based on multi-scale cross-attention is designed to integrate deep features;

[0060] The features of each layer are input into the decoder and connected in series with the features of the previous layer in the channel dimension to restore the fused image I f ∈R H×W×3 , the formula is:

[0061]

[0062] in represents the i-th layer of the decoder. When i=5, the decoder input is the fused features of the fifth layer. When i=1, 2, 3, 4, the decoder input of the i-th layer is the result of the concatenation of the fused features of the i-th layer and the output of the previous layer of decoder in the channel dimension. Represents the feature aggregation of the i-th layer. When i=1, the decoder outputs the fused image I fu .

[0063] In some implementations, the design of the gradient enhancement module at the middle level is specifically:

[0064] The mixed features of the input gradient enhancement module are first passed through a layer of convolution, and then the Sobel operator, Laplacian operator and Prewitt operator are used to calculate the gradient of the features output by the first layer of convolution. The Sobel operator is the Sobel operator, the Laplacian operator is the Laplace operator, and the Prewitt operator is the Prewitt operator. Then, the gradient features calculated by the three gradient operators are connected in series in the channel dimension and sent to the convolution layer, and then connected in series with the mixed features of the initial input in the channel dimension and sent to the convolution layer. The layer formula is:

[0065]

[0066] where fu i represents the output of the gradient enhancement module, Conv(·) represents the convolution operation, j = 1, 2, 3, and They represent Sobel operator, Laplacian operator and Prewitt operator respectively; It is a middle-level mixed feature connected in series in the channel dimension.

[0067] In some implementations, the multi-scale semantic interaction module is:

[0068] Three convolutional layers with different kernel sizes are used to capture multi-scale deep features, and then the features at different scales are concatenated in the channel dimension. The formula is:

[0069]

[0070] represents the result of element-by-element addition of features of different scales of the i-th layer x modality, where x∈{ir,vi} represents the modality, add(·) represents element-by-element addition, and Conv n×n Represents convolution operations of different scales, n=7, 11, 21, Represents the features of mode x at layer i;

[0071] Deploy a projection function containing a convolution kernel reshaping operation to convert multi-scale features into keys and values. The formula is:

[0072]

[0073] represents a key, Representative value; Conv 1×1(·) and Reshape(·) correspond to the convolutional layer with kernel size 1×1 and the reshape operation, respectively; H i , W i and C i Refers to the input features The height, width and channels of the infrared and visible light features are:

[0074]

[0075] in Represents features of another modality;

[0076] Calculate the modal attention map corresponding to each modality x The formula is:

[0077]

[0078] Among them, Softmax(·) represents the Soft max function, represents the transpose of the i-th level feature key of the mode of x;

[0079] Utilize Value Multiply the corresponding attention to obtain features with global context, then reshape the global features, and use 1×1 convolution to adjust the channels and connect them in the channel dimension to generate fused features rich in semantics. The formula is:

[0080]

[0081] In some embodiments, the constructed loss function guides the training of the hierarchical infrared-visible light image fusion network by using a weighted combination of the global fusion loss and the local information supplementary fusion loss to obtain the total loss of the fusion network, and the formula is:

[0082] L Local =L Holistic +μ·L Local

[0083] μ is a hyperparameter.

[0084] In some embodiments, the global fusion loss is: an intensity loss, a texture loss, and a style loss are introduced to limit the visual quality of the fusion result;

[0085] Among them, for daytime conditions, the light source is single, and using the intensity loss with the maximum pixel value is reasonable for extracting salient areas from the two modalities;

[0086] For nighttime conditions, due to the presence of local area exposure, the mask generated by the mask modeling algorithm is used to construct the intensity loss at night. The scene probability output by the SAN is used to reasonably guide the network to avoid the local light source exposure area in the visible light image, and retain the salient targets and textures in the area from the infrared modality.

[0087] Therefore, the strength loss can be expressed as:

[0088]

[0089] in represents the intensity loss used in night scenes, represents the intensity loss used in daytime scenes; p P and P n is the output of SAN, representing the probability of the visible light image belonging to daytime and nighttime, w ir and w vi YesP d and P n The normalized result;

[0090]

[0091] max(·) represents the maximum value operation, ||·||1 is the L1 norm, ||·||2 is the L2 norm, I f represents the fused image, I ir and I vi Represent infrared image and visible light image respectively, I M The mask generated by Algorithm 1;

[0092] In addition, in order to make the fused image maintain the best intensity distribution in various scenes while retaining rich texture details; the maximum aggregation of infrared and visible light image textures is used to constrain the fused image to contain rich texture information; therefore, the texture loss is defined as follows:

[0093]

[0094] represents the Sobel operator;

[0095] Inspired by the style transfer task, a style control loss function is designed to prevent the style of the fusion result from being biased towards a single modality:

[0096]

[0097] Where G represents the Gram matrix;

[0098] The global fusion loss is a weighted combination of intensity loss, texture loss, and style loss as follows:

[0099] L Holistic =λ1·L int +λ2·L text +λ3·L Gram

[0100] Among them, λ1, λ2, and λ3 are hyperparameters.

[0101] In some embodiments, the local information supplementation fusion loss is:

[0102] Considering the serious information loss in the local exposure area of ​​the visible light image, the local information supplement fusion loss is used to guide the network to integrate the information from the infrared modality into the fused image;

[0103] The local information complementary fusion loss is defined as:

[0104]

[0105] Specifically, the scene-aware network and mask modeling algorithm are used to construct global intensity and local intensity loss functions to maximize the use of detail information in any scene; in order to maintain the fine-grained details of the original image in the fusion result, it is necessary for the fusion network to maintain the full resolution of the input information throughout the process, and image-level fusion has to rely on downsampling to capture information of different scales in order to provide sufficient semantic information for subsequent tasks, so a U-shaped network design is adopted; in order to supplement the details as much as possible and reduce information loss, FMEM is used to reuse the shallow information of the network and maximize the use of the full-resolution information of the image; in the middle layer of the network, GEM is designed to enhance the bimodal texture features in the mixed features; in the deep layer of the network, MSIM is used to capture deep semantic features to supplement the fusion results with sufficient semantic information.

[0106] Specifically, in the infrared and visible light image fusion task in this application, for different lighting scenes, the scene perception network and mask modeling algorithm are first used to construct the fusion loss; the gradient enhancement module is designed to enhance the texture information in the mixed modality at the feature level; considering the need for semantic information in high-level visual tasks, a multi-scale semantic interaction module is designed at the deep semantic level to correctly utilize the semantic information in the bimodal features; finally, a large number of experiments show that the method of this application outperforms the most advanced algorithms in terms of visual quality and quantitative evaluation, and promotes subsequent high-level visual tasks.

[0107] Specifically, in order to verify the results of the infrared visible light image fusion method of the present application, the effectiveness is verified by simulation as follows:

[0108] First, in one embodiment of the present application, the infrared image and visible light image data in the MSRS data set are fused by the infrared-visible light image fusion method for complex lighting environments proposed by the present invention, and the specific process is as follows:

[0109] We train the infrared and visible light image fusion method for complex lighting environments on the training set of the MSRS dataset. We select 715 daytime images and 729 nighttime images to train our scene perception network. Each image is cropped and multiplied by four equal parts of 320*240. Therefore, there are 2916 daytime images and 2840 nighttime images after expansion. We use one-hot labels, (1, 0) and (0, 1) to represent daytime and nighttime images respectively. The training set and test set are divided into 9:1. The network optimizer uses Adam, epoch=100, batchsize=32, the learning rate is 1*10-3 and it is turned on at the halfway point of training. The fusion network is trained on the training set of MSRS. The network optimizer uses Adam, epoch=30, batchsize=8, and the learning rate setting is the same as that of the scene perception network. The loss function parameters are λ1=30, λ2=20, λ3=50, and μ=100. The optimizer Adam has the characteristics of small memory usage and is suitable for processing sparse gradients. It is currently widely used in various fields and has good performance. The test uses the public data sets MSRS, LLVIP, and M3FD test set parts, and selects 200, 40, and 20 pairs of images for algorithm comparison experiments. All comparison algorithms in the experiment are tested with the settings of the original paper. The entire network is trained using the Pytorch framework on NVIDIA GeForce RTX4090 and Intel Core i9-13900K.

[0110] Specifically, the effect analysis of the simulated experimental image fusion results is carried out:

[0111] It is compared with the other seven state-of-the-art image fusion algorithms, namely BTSFusion, CoCoNet, CSF, DDcGAN, DDFM, DIVFusion and RFN-Nest. Figure 4 It is the comparative experimental result of different methods on images with complex lighting conditions in simulation experiments. Figure 5 This is a comparison of the semantic segmentation results of the fusion results of different algorithms; Figure 4 The fusion result images of seven comparison methods are shown in Figure 2. From the perspective of detail extraction, since the visible light sensor relies on light reflection imaging, Figure 4In complex lighting conditions, image detail information is severely degraded. When encountering strong direct light, the information of the local exposure area is completely lost. Among them, infrared sensors capture thermal radiation imaging and will not be polluted by strong light. Infrared images contain significant targets and extensive texture details, which can supplement the texture degradation and detail loss problems of visible images.

[0112] However, it is a challenge to adaptively integrate the texture information of infrared and visible light images according to different scenes. Figure 4 In the scene, DDcGAN causes blurring of textures and salient targets, BTSFusion, CSF, DDFM, and RFN-Nest all weaken salient targets to varying degrees, and only DIVFusion, CoCoNet, and the method of this application reasonably maintain the strength of salient targets. However, when processing the information of the exposure area, only the method of this application can reasonably utilize the detailed texture information between the two modalities and maximize the retention of detailed information in the fusion result; only the method of this application can retain the intensity of salient targets while reasonably retaining the detailed information of the exposure area. It can be seen that the method of this article has a strong ability to retain details in image fusion.

[0113] In order to verify the method's ability to promote high-level visual tasks, the fusion results were tested using DeeplabV3+. The comparison results with other methods are shown in the figure below. Figure 5 As shown in the figure, since DeeplabV3+ is trained on a visible image dataset, the fusion result with an overall style biased towards the visible light modality will achieve better segmentation performance; this phenomenon can be discovered by focusing on the sky in the scene. The method of this application makes reasonable use of the dual-modal features, and the segmentation model identifies more sky areas. In addition, the practical fusion method can integrate the semantic information of the source image to improve the ability of the segmentation model to understand the scene, which proves the advantage of the method of this application in reasonably integrating semantic information.

[0114] Specifically, visual evaluation and comparison can give a more intuitive understanding of the image fusion results, but it is difficult to give the most accurate judgment of the image fusion results by relying solely on subjective evaluation. Therefore, it is necessary to combine objective indicators to evaluate the image fusion results. This experimental design includes multiple key performance indicators to evaluate and quantify the effectiveness of the model. Six evaluation indicators are selected for quantitative evaluation. The evaluation results are shown in Table 1, including mutual information MI, sum of difference correlation SCD, visual information fidelity VIF, edge information-based indicator Qabf, structural similarity metric SSIM and multi-scale structural similarity metric MS-SSIM. Among them, the MI metric measures the similarity of the source image from the perspective of information theory. The SCD index characterizes the quality of the fusion algorithm by measuring the difference between the fused image and the source image. The VIF index evaluates the information fidelity of the fused image from the perspective of the human visual system. Qabf measures the amount of edge information transmitted from the source image to the fused image. SSIM models the information loss and distortion in the fusion process, reflecting the structural similarity between the fused image and the source image. MS-SSIM comprehensively evaluates the distortion of the fused image based on SSIM and combines the structural similarity at multiple scales. In addition, the larger the MI, SCD, VIF, Qabf, SSIM and MS-SSIM, the better the fusion performance.

[0115] Table 1 Index evaluation results of different methods

[0116]

[0117]

[0118] Specifically, Table 1 lists in detail the comparative data of the present application method and several other algorithms in various evaluation indicators, thereby comprehensively demonstrating the performance and advantages of the present application method; the present application method has achieved great improvement in various indicators, which shows that the method of this article can ensure the image quality of the fusion result; comprehensive visual evaluation and objective indicator evaluation, the infrared and visible light image fusion method for complex lighting environments given in this embodiment can integrate detail information from the source images of the two modalities to produce a fusion result with rich details.

[0119] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. The above description is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and variations can be made without departing from the technical principles of the present invention. These improvements and variations should also be regarded as the protection scope of the present invention.

Claims

1. A method for fusion of infrared and visible light images in complex lighting environments, characterized in that: The following steps are involved: S1: Acquire multiple pairs of infrared images and visible light images after registration in various scenes to form training image subsets to obtain training data sets, and identify the training data sets using a scene perception network; S2: Input a subset of training images identified as nighttime scenes into a mask modeling algorithm to generate masks to guide network training; S3: Constructing a hierarchical infrared and visible light image fusion network; S4: Construct a loss function to guide the training of the hierarchical infrared-visible image fusion network.

2. The infrared and visible light image fusion method for complex lighting environments according to claim 1, characterized in that: In S1, the scene perception network includes: a convolutional layer, a batch normalization layer, a ReLu activation function, a maximum pooling layer and a fully connected layer; The training data set is identified by using a scene perception network as follows: outputting the probability that the training data set belongs to a daytime scene and a nighttime scene.

3. The infrared and visible light image fusion method for complex lighting environments according to claim 1, characterized in that: In S2, the mask modeling algorithm is: input the visible light image in the training image subset into the scene perception network for identification, perform Gaussian filtering on the image identified as a night scene, binarize the result of Gaussian filtering, process the binarized result, and then superimpose the processed binarized result with the character semantic segmentation label to generate a mask to guide network training.

4. The infrared and visible light image fusion method for complex lighting environments according to claim 1, characterized in that: The hierarchical fusion network includes: a shallow feature fusion layer, a middle gradient enhancement layer and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module FMEM for enhanced extraction of shallow features, the middle gradient enhancement layer includes a gradient enhancement module GEM for gradient enhancement of middle mixed features, and the deep semantic interaction layer includes a multi-scale semantic interaction module MSIM for integrating deep features.

5. The infrared and visible light image fusion method for complex lighting environments according to claim 4 is characterized in that: The hierarchical infrared-visible light image fusion network constructed in S3 is: Input the infrared image and visible light image of the training data set into the encoder of the network model to extract features, extract the features of the i-th layer respectively, and then input the features of the infrared image and visible light image in the shallow layer into the feature reuse module after being concatenated in the channel dimension for further feature extraction and reuse; A gradient enhancement module is designed in the middle layer to enhance the gradient texture features at the mixed feature level, connecting the infrared image and visible light image features in series in the channel dimension. In the deep layer, a multi-scale semantic interaction module based on multi-scale cross-attention is designed to integrate deep features; The features of each layer are input into the decoder and connected in series with the features of the previous layer in the channel dimension to restore the fused image.

6. The infrared and visible light image fusion method for complex lighting environments according to claim 5, characterized in that: The gradient enhancement module designed in the middle layer is specifically as follows: The mixed features of the input gradient enhancement module are first passed through a layer of convolution, and then the Sobel operator, Laplacian operator and Prewitt operator are used to calculate the gradient of the features output by the first layer of convolution respectively. The gradient features calculated by the three gradient operators are then connected in series in the channel dimension and sent to the convolution layer, and then connected in series with the mixed features of the initial input in the channel dimension and sent to the convolution layer.

7. The infrared and visible light image fusion method for complex lighting environments according to claim 5, characterized in that: The multi-scale semantic interaction module is: Three convolutional layers with different kernel sizes are used to capture multi-scale deep features. The features at different scales are then concatenated in the channel dimension. A projection function containing a convolution kernel reshaping operation is deployed to convert the multi-scale features into keys and values. The modal attention map corresponding to each modality x is calculated, and the value is multiplied by the corresponding attention to obtain features with global context. The global features are then reshaped, and a 1×1 convolution is used to adjust the channel and then connected in the channel dimension to generate a fused feature rich in semantics.

8. The infrared and visible light image fusion method for complex lighting environments according to claim 1, characterized in that: The constructed loss function guides the training of the hierarchical infrared-visible light image fusion network by using a weighted combination of the global fusion loss and the local information supplementary fusion loss to obtain the total loss of the fusion network.

9. The infrared and visible light image fusion method for complex lighting environments according to claim 8, characterized in that: The global fusion loss is: intensity loss, texture loss, style loss; Among them, for daytime conditions, the light source is single, and using the intensity loss with the maximum pixel value is reasonable for extracting salient areas from the two modalities; For nighttime situations, due to the existence of local area exposure, the mask generated by the mask modeling algorithm is used to construct the intensity loss at night. The scene probability output by the SAN is used to reasonably guide the network to avoid the local light source exposure area in the visible light image and retain the salient targets and textures in the area from the infrared modality.

10. The infrared and visible light image fusion method for complex lighting environments according to claim 8, characterized in that: The local information supplementary fusion loss is: Considering the serious information loss in the local exposure area of ​​the visible light image, the local information supplement fusion loss is used to guide the network to integrate the information from the infrared modality into the fused image.

Images