An infrared and visible light image fusion method for complex lighting environment

Abstract

The application provides an infrared and visible light image fusion method under a complex lighting environment, which comprises the following steps: obtaining a plurality of pairs of registered infrared images and visible light images under a plurality of scenes to form a training image subset, so as to obtain a training data set, and identifying the training data set by using a scene perception network; inputting the training image subset identified as a night scene into a mask modeling algorithm to generate a mask for guiding the training of a network; constructing a hierarchical infrared and visible light image fusion network; and constructing a loss function to guide the training of the hierarchical infrared and visible light image fusion network. According to the application, the fusion loss is constructed by using the scene perception network and the mask modeling algorithm to adapt to different lighting scenes; the gradient enhancement module is designed to enhance the texture information in the mixed modalities at the feature level; and the multi-scale semantic interaction module is introduced at the deep semantic level to effectively utilize the semantic information in the dual modal features.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an infrared-visible light image fusion method for complex lighting environments. Background Technology

[0002] Currently, images captured by single-modal sensors cannot effectively and comprehensively describe the imaging scene; while visible light sensors, through reflected light imaging, can provide high spatial resolution background details, but cannot clearly see targets when lighting or camouflage conditions are poor. In contrast, infrared sensors image based on the differentiated thermal radiation emitted by objects, are unaffected by adverse conditions, and can operate in all weather conditions. This complementary nature has led to the development of infrared-visible light image fusion, where the fused image can highlight targets and reveal rich texture details. Therefore, infrared and visible light image fusion has been widely used as a preprocessing module for advanced vision tasks, such as object detection, person re-identification, and semantic segmentation.

[0003] Existing methods are often designed for conditions with uniform illumination. Infrared and visible light image fusion may face various scenarios, including day and night. In daytime scenes, the light source is uniformly distributed, and the visible light image contains more meaningful information. In nighttime scenes, the light source is extremely unevenly distributed, and there is a common situation where the brightness of the light area is too high, i.e., local exposure. When faced with point local exposure, existing methods cannot make reasonable use of the detailed information of the local area, resulting in serious loss of detail and failing to generate fused images that conform to human vision and are beneficial to downstream advanced vision tasks. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the defects mentioned above, thereby providing an infrared and visible light image fusion method for complex lighting environments.

[0005] To address the aforementioned problems, this invention provides an infrared-visible light image fusion method for complex lighting environments, comprising the following steps:

[0006] S1: Obtain multiple pairs of registered infrared and visible light images under various scenarios to form a training image subset, thereby obtaining a training dataset, and then use a scene-aware network to label the training dataset;

[0007] S2: Input a subset of training images identified as nighttime scenes into the mask modeling algorithm to generate a mask to guide network training;

[0008] S3: Construct a hierarchical infrared-visible image fusion network;

[0009] S4: Construct a loss function to guide the training of a hierarchical infrared-visible image fusion network.

[0010] Preferably, in S1, the scene-aware network includes: a convolutional layer, a batch normalization layer, a ReLU activation function, a max pooling layer, and a fully connected layer;

[0011] The training dataset is identified using a scene-aware network to output the probability that the training dataset belongs to a daytime scene or a nighttime scene.

[0012] Preferably, in step S2, the mask modeling algorithm is as follows: the visible light images in the training image subset are input into the scene perception network for identification; the images identified as night scenes are subjected to Gaussian filtering; the Gaussian filtering result is binarized; the binarized result is processed; and the processed binarized result is superimposed with the person semantic segmentation label to generate a mask to guide the network training.

[0013] Preferably, the hierarchical fusion network includes: a shallow feature fusion layer, a mid-level gradient enhancement layer, and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module FMEM for enhancing shallow features, the mid-level gradient enhancement layer includes a gradient enhancement module GEM for enhancing mid-level hybrid features, and the deep semantic interaction layer includes a multi-scale semantic interaction module MSIM for integrating deep features.

[0014] Preferably, the hierarchical infrared-visible image fusion network constructed in S3 is as follows:

[0015] Infrared and visible light images from the training dataset are input into the encoder of the network model to extract features. Features from layer i are extracted separately. The features from the shallow infrared and visible light images are concatenated along the channel dimension and then input into the feature reuse module for further feature extraction and reuse.

[0016] A gradient enhancement module is designed in the middle layer to enhance gradient texture features at the level of mixed features, concatenating infrared and visible light image features in the channel dimension;

[0017] A multi-scale semantic interaction module based on multi-scale cross-attention is designed to integrate deep features;

[0018] Each layer of features is input into the decoder and concatenated with the features of the previous layer in the channel dimension to restore the fused image.

[0019] Preferably, the gradient enhancement module designed in the middle layer specifically refers to:

[0020] The mixed features of the input gradient enhancement module are first passed through a convolutional layer. Then, the Sobel operator, Laplacian operator, and Prewitt operator are used to calculate the gradients of the features output by the first convolutional layer. The gradient features calculated by the three gradient operators are then concatenated along the channel dimension and fed into the convolutional layer. Finally, the mixed features of the initial input are concatenated along the channel dimension and fed into the convolutional layer.

[0021] Preferably, the multi-scale semantic interaction module is:

[0022] Three convolutional layers with different kernel sizes are used to capture multi-scale deep features. Then, the features at different scales are concatenated along the channel dimension. A projection function containing convolutional kernel reshaping operation is deployed to convert the multi-scale features into keys and values. The modality attention map corresponding to each modality x is calculated. The value is multiplied by the corresponding attention to obtain features with global context. Then, the global features are reshaping operation, and 1×1 convolution is used to adjust the channels and then concatenated along the channel dimension to generate fusion features rich in semantics.

[0023] Preferably, the construction of the loss function to guide the training of the hierarchical infrared and visible light image fusion network is as follows: a weighted combination of global fusion loss and local information-supplemented fusion loss is used to obtain the total loss of the fusion network.

[0024] Preferably, the global fusion loss is: intensity loss, texture loss, and style loss;

[0025] In the case of daytime, where the light source is singular, using the intensity loss with the highest pixel value is reasonable for extracting significant regions from both modalities.

[0026] For nighttime scenarios, due to localized exposure issues, a mask modeling algorithm is used to generate a mask to construct the nighttime intensity loss. The scene probability output by SAN is used to guide the network to avoid localized light source exposure areas in the visible light image, while retaining salient targets and textures within the area from the infrared mode.

[0027] Preferably, the local information supplementation and fusion loss is:

[0028] Considering the severe information loss in locally exposed areas of visible light images, local information is used to supplement the fusion loss to guide the network in integrating information from infrared modes into the fused image.

[0029] The infrared-visible light image fusion method provided by this invention for complex lighting environments has the following beneficial effects:

[0030] This invention constructs a fusion loss using scene-aware networks and mask modeling algorithms to adapt to different lighting scenarios; it also enhances texture information in the hybrid modality at the feature level by designing a gradient enhancement module; and it effectively utilizes semantic information in the dual-modal features by introducing a multi-scale semantic interaction module at the deep semantic level. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the infrared and visible light image fusion method framework of the present invention;

[0032] Figure 2 This is a schematic diagram of the gradient enhancement module structure of the present invention;

[0033] Figure 3 This is a schematic diagram of the multi-scale semantic interaction module structure of the present invention;

[0034] Figure 4 This is a schematic diagram showing the comparison results of the simulation experiment of the present invention;

[0035] Figure 5 This is a schematic diagram showing the complete results of semantic segmentation using DeepLabV3+ in the simulation experiment of this invention. Detailed Implementation

[0036] like Figure 1-5 As shown, this invention provides an infrared-visible light image fusion method for complex lighting environments, which includes the following steps:

[0037] S1: Obtain multiple pairs of registered infrared and visible light images under various scenarios to form a training image subset, thereby obtaining a training dataset, and then use a scene-aware network to label the training dataset;

[0038] S2: Input a subset of training images identified as nighttime scenes into the mask modeling algorithm to generate a mask to guide network training;

[0039] S3: Construct a hierarchical infrared-visible image fusion network;

[0040] S4: Construct a loss function to guide the training of a hierarchical infrared-visible image fusion network.

[0041] In some implementations, the scene-aware network in S1 includes: a convolutional layer, a batch normalization layer, a ReLU activation function, a max pooling layer, and a fully connected layer;

[0042] The training dataset is identified using a scene-aware network to output the probability that the training dataset belongs to a daytime scene or a nighttime scene.

[0043] In some implementations, in step S2, the mask modeling algorithm is as follows: Visible light images from the training image subset are input into a scene perception network for labeling; images labeled as nighttime scenes are subjected to Gaussian filtering; regions with pixel values ​​greater than 300 and cumulative pixel counts greater than 70 are set to 1, and the rest to 0; the Gaussian filtering result is binarized; regions with pixel values ​​greater than 300 and cumulative pixel counts greater than 70 are set to 1, and the rest to 0; after processing the binarization result, the processed binarization result is superimposed with the person semantic segmentation label to generate a mask M to guide network training.

[0044] Specifically, the processing includes: performing morphological erosion and dilation on the binarized results to eliminate noise.

[0045] In some implementations, the hierarchical fusion network includes: a shallow feature fusion layer, a mid-level gradient enhancement layer, and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module (FMEM) for enhancing shallow features, the mid-level gradient enhancement layer includes a gradient enhancement module (GEM) for enhancing mid-level hybrid features, and the deep semantic interaction layer includes a multi-scale semantic interaction module (MSIM) for integrating deep features.

[0046] In some implementations, the hierarchical infrared-visible image fusion network constructed in S3 is as follows:

[0047] Infrared images from the training dataset I ir ∈R H×W×1 and visible light image I vi ∈R H×W×3 The encoder of the input network model extracts features. H×W×1 represents an infrared image with height H, width W, and 1 channel, and H×W×3 represents a visible light image with height H, width W, and 3 channels. The formula is:

[0048] {F ir F vi}={E(I ir ), E(I vi )}

[0049] Where F ir and F vi Indicates infrared and visible light characteristics, E(·) represents the encoder;

[0050] Extract the features of layer i respectively, and and The features extracted from the infrared and visible light images by the i-th layer of the network are defined respectively, and the formula is:

[0051]

[0052] in Represents the infrared and visible light features extracted by the network at layer i. and Let represent the infrared feature extraction block and the visible light feature extraction block of the i-th layer of the encoder, where i = 1, 2, 3, 4, 5. When i = 1, and The input is a visible light image and a visible light image. When i = 2, 3, 4, 5, and The input consists of features from the two modalities of the previous layer;

[0053] Features from shallow infrared and visible light images are concatenated along the channel dimension and then input into the feature reuse module for further feature extraction and reuse. The formula is as follows:

[0054]

[0055] The concatenated features are represented by concat(·), which indicates concatenation along the channel dimension. fu1 represents the fused features obtained from the first layer of the network, and FMEM{·} represents the feature reuse extraction module.

[0056] A gradient enhancement module is designed in the middle layer to enhance gradient texture features at the hybrid feature level, concatenating infrared and visible light image features along the channel dimension, as shown in the formula:

[0057]

[0058] Where i = (2, 3) represents the features of the i-th layer. This represents the result after concatenation along the channel dimension;

[0059] A multi-scale semantic interaction module based on multi-scale cross-attention is designed to integrate deep features;

[0060] Each layer's features are input into the decoder and concatenated sequentially with the features from the previous layer along the channel dimension to reconstruct the fused image I. f ∈R H×W×3 The formula is:

[0061]

[0062] in This represents the i-th layer of the decoder. When i = 5, the decoder input is the fused feature of the fifth layer. When i = 1, 2, 3, 4, the decoder input for the i-th layer is the result of concatenating the fused feature of the i-th layer with the output of the previous layer along the channel dimension. This represents the feature aggregation of the i-th layer. When i=1, the decoder outputs the fused image I. fu .

[0063] In some implementations, the gradient enhancement module designed in the middle layer specifically refers to:

[0064] The mixed features input to the gradient enhancement module are first passed through a convolutional layer. Then, the Sobel, Laplacian, and Prewitt operators are used to calculate the gradients of the features output from the first convolutional layer. The gradient features calculated by the three gradient operators are then concatenated along the channel dimension and fed into the convolutional layer. Finally, the mixed features from the initial input are concatenated along the channel dimension and fed into the convolutional layer. The layer formula is as follows:

[0065]

[0066] Among them fu i Represents the output of the gradient enhancement module, Conv(·) denotes the convolution operation, j = 1, 2, 3, and These represent the Sobel operator, the Laplacian operator, and the Prewitt operator, respectively. It is a mid-level hybrid feature that is connected in series along the channel dimension.

[0067] In some implementations, the multi-scale semantic interaction module is:

[0068] Three convolutional layers with different kernel sizes are used to capture multi-scale depth features. These features from different scales are then concatenated along the channel dimension, as shown in the formula:

[0069]

[0070] This represents the result of element-wise addition of different scale features of the i-th layer x-mode, where x∈{ir, vi} represents the mode, add(·) represents element-wise addition, and Conv n×n These represent convolution operations at different scales, n = 7, 11, 21. This represents the characteristics of the x-mode at layer i;

[0071] Deploy a projection function that includes a convolutional kernel reshaping operation to convert multi-scale features into keys and values, as shown in the following formula:

[0072]

[0073] Representative key, Representative value; Conv 1×1(·) and Reshape(·) correspond to a convolutional layer with a kernel size of 1×1 and a reshaping operation, respectively; H i W i and C i These refer to the input features respectively. The height, width, and channels; it should be noted that infrared and visible light features mutually generate queries for each other's modalities, as shown in the equation; this allows infrared and visible light features to mutually guide feature extraction in depth and heterogeneous spaces, enabling the network to fully utilize the complementary properties of multimodal features:

[0074]

[0075] in This represents a feature of another mode;

[0076] Calculate the modal attention map corresponding to each modality x. The formula is:

[0077]

[0078] Where Softmax(·) represents the Soft max function, The transpose of the i-th layer feature bond of the x-mode;

[0079] Utilization value Multiply by the corresponding attention to obtain features with global context. Then, perform a reshaping operation on the global features and use 1×1 convolutions to adjust the channels before concatenating them along the channel dimension to generate semantically rich fused features. The formula is as follows:

[0080]

[0081] In some implementations, the construction of the loss function to guide the training of the hierarchical infrared-visible image fusion network is as follows: a weighted combination of global fusion loss and local information-supplemented fusion loss is used to obtain the total loss of the fusion network, and the formula is:

[0082] L Local =L Holistic +μ·L Local

[0083] μ is a hyperparameter.

[0084] In some implementations, the global fusion loss is: intensity loss, texture loss, and style loss are introduced to limit the visual quality of the fusion result;

[0085] In the case of daytime, where the light source is singular, using the intensity loss with the highest pixel value is reasonable for extracting significant regions from both modalities.

[0086] For nighttime scenarios, due to localized exposure issues, a mask modeling algorithm is used to generate a mask to construct the intensity loss at night. The scene probability output by SAN is used to guide the network to avoid localized light source exposure areas in the visible light image, and to preserve salient targets and textures within the area from the infrared mode.

[0087] Therefore, the strength loss can be expressed as:

[0088]

[0089] in Intensity loss representing nighttime use. This represents the intensity loss during daytime use; p P and P n The output of SAN represents the probability that a visible light image belongs to daytime or nighttime, w. ir and w vi It is P d and P n The result after normalization;

[0090]

[0091] `max(·)` represents the maximum value operation. `||·||1` is the L1 norm, `||·||2` is the L2 norm, and I... f Represents the fused image, I ir and I vi Representing infrared and visible light images respectively, I M The mask generated by Algorithm 1;

[0092] Furthermore, to ensure the fused image maintains optimal intensity distribution across various scenes while preserving rich texture details, the maximum aggregation of infrared and visible light image textures is used to constrain the fused image to contain rich texture information; therefore, the texture loss is defined as follows:

[0093]

[0094] Represents the Sobel operator;

[0095] Inspired by style transfer tasks, a style control loss function was designed to prevent the fusion result from being biased towards a single modality.

[0096]

[0097] Where G represents the Gram matrix;

[0098] The global fusion loss is a weighted combination of intensity loss, texture loss, and style loss, as follows:

[0099] L Holistic =λ1·L int +λ2·L text +λ3·L Gram

[0100] Where λ1, λ2, and λ3 are hyperparameters.

[0101] In some implementations, the local information supplementation and fusion loss is:

[0102] Considering the severe information loss in local exposure areas of visible light images, local information is used to supplement the fusion loss to guide the network to integrate information from infrared modes into the fused image;

[0103] The definition of local information supplementation fusion loss is:

[0104]

[0105] Specifically, scene-aware networks and mask modeling algorithms are used to construct global and local intensity loss functions to maximize the utilization of detail information in any scene. To preserve the fine-grained details of the original image in the fusion result, it is necessary for the fusion network to maintain the full resolution of the input information throughout the process. However, image-level fusion must rely on downsampling to capture information at different scales in order to provide sufficient semantic information for subsequent tasks, so a U-shaped network design is adopted. In order to supplement details as much as possible and reduce information loss, FMEM is used to reuse the shallow information of the network and maximize the utilization of the full resolution information of the image. In the middle layer of the network, GEM is designed to enhance the bimodal texture features in the mixed features. In the deep layer of the network, MSIM is used to capture deep semantic features and supplement sufficient semantic information for the fusion result.

[0106] Specifically, in this application, for the infrared-visible light image fusion task, for different lighting scenarios, a scene-aware network and mask modeling algorithm are first used to construct the fusion loss; a gradient enhancement module is designed to enhance the texture information in the mixed modality at the feature level; considering the semantic information requirements of advanced vision tasks, a multi-scale semantic interaction module is designed at the deep semantic level to correctly utilize the semantic information in the bimodal features; finally, extensive experiments show that the method in this application outperforms state-of-the-art algorithms in terms of visual quality and quantitative evaluation, and promotes subsequent advanced vision tasks.

[0107] Specifically, to verify the results of the infrared-visible light image fusion method of this application, the effectiveness of the method is verified through simulation below:

[0108] First, in one embodiment of this application, the infrared and visible light image data in the MSRS dataset are fused using the infrared and visible light image fusion method for complex lighting environments proposed in this invention. The specific process is as follows:

[0109] We trained an infrared-visible image fusion method for complex lighting environments on the MSRS dataset training set. We selected 715 daytime images and 729 nighttime images to train our scene perception network. Each image was cropped into four equal parts of 320*240 pixels, resulting in 2916 augmented daytime images and 2840 augmented nighttime images. One-hot labels were used, with (1,0) and (0,1) representing daytime and nighttime images respectively. The training and test sets were split in a 9:1 ratio. The network optimizer used was Adam, with epoch = 100, batch size = 32, and a learning rate of 1*10⁻³. The network was optimized during the half-training phase. The network initially exhibits linear decay. A fusion network is trained on the MSRS training set using the Adam optimizer. The epochs are 30, the batch size is 8, and the learning rate is the same as that of the scene-aware network. The loss function parameters are λ1 = 30, λ2 = 20, λ3 = 50, and μ = 100. The Adam optimizer is characterized by its small memory footprint and suitability for handling sparse gradients, making it a widely used and high-performing optimization algorithm across various fields. Testing is conducted using portions of the public datasets MSRS, LLVIP, and M3FD, with 200, 40, and 20 pairs of images selected for each dataset for comparative experiments. All comparison algorithms are tested using the settings described in the original paper. The entire network is trained using the PyTorch framework on an NVIDIA GeForce RTX 4090 and an Intel Core i9-13900K.

[0110] Specifically, the effect analysis of the image fusion results in the simulation experiment is performed:

[0111] It is compared with seven other state-of-the-art image fusion algorithms, namely BTSFusion, CoCoNet, CSF, DDcGAN, DDFM, DIVFusion, and RFN-Nest. Figure 4 These are the comparative experimental results of different methods applied to images under complex lighting conditions in a simulation experiment. Figure 5 This is a comparison image of semantic segmentation results from the fusion of different algorithms; in Figure 4 The image shown is the fusion result of seven comparison methods. From the perspective of detail extraction, since visible light sensors rely on light reflection for imaging, in... Figure 4Medium lighting conditions are complex, resulting in severe degradation of image detail. When exposed to direct strong light, information in locally exposed areas is completely lost. In contrast, infrared sensors capture thermal radiation imaging, which is not affected by strong light. Infrared images contain prominent targets and extensive texture details, thus compensating for the texture degradation and detail loss problems of visible images.

[0112] However, adaptively integrating texture information from infrared and visible light images according to different scenarios is a challenge. Figure 4 In the scene, DDcGAN causes blurring of textures and salient objects. BTSFusion, CSF, DDFM, and RFN-Nest all weaken salient objects to varying degrees. Only DIVFusion, CoCoNet, and the method in this application reasonably maintain the intensity of salient objects. However, when processing information in the exposure area, only the method in this application can reasonably utilize the detailed texture information between the two modalities and maximize the preservation of detailed information in the fusion result. Only the method in this application can reasonably preserve the detailed information of the exposure area while preserving the intensity of salient objects. It can be seen that the method in this application has a strong ability to preserve details in image fusion.

[0113] To test the method's ability to facilitate advanced vision tasks, the fusion results were tested using DeeplabV3+, and the comparison results with other methods are as follows: Figure 5 As shown, since DeeplabV3+ is trained on a visible image dataset, the fusion results with an overall style bias towards the visible light modality will achieve better segmentation performance. This phenomenon can be observed by focusing on the sky in the scene; the method in this application makes reasonable use of bimodal features, and the segmentation model identifies more sky regions. Furthermore, the practical fusion method can integrate the semantic information of the source images to improve the segmentation model's ability to understand the scene, demonstrating the advantage of the method in reasonably integrating semantic information.

[0114] Specifically, visual evaluation comparison can provide a relatively intuitive understanding of the image fusion results, but relying solely on subjective evaluation makes it difficult to give the most accurate judgment of the image fusion results. Therefore, it is necessary to combine objective indicators to jointly evaluate the image fusion results. This experimental design includes multiple key performance indicators to evaluate and quantify the effectiveness of the model. Six evaluation indicators were selected for quantitative evaluation, and the evaluation results are shown in Table 1. These include mutual information (MI), sum of difference correlations (SCD), visual information fidelity (VIF), edge information-based indicator (Qabf), structural similarity measure (SSIM), and multi-scale structural similarity measure (MS-SSIM). Among them, the MI metric measures the difference in image fusion results from the perspective of information theory. The metrics include: the amount of information transmitted to the fused image; the SCD metric, which measures the difference between the fused image and the source image to characterize the quality of the fusion algorithm; the VIF metric, which evaluates the information fidelity of the fused image from the perspective of the human visual system; the Qabf metric, which measures the amount of edge information transmitted from the source image to the fused image; the SSIM metric, which models information loss and distortion during the fusion process and reflects the structural similarity between the fused image and the source image; and the MS-SSIM metric, which, based on the SSIM metric, combines structural similarity at multiple scales to comprehensively evaluate the distortion of the fused image. In addition, the larger the values ​​of MI, SCD, VIF, Qabf, SSIM, and MS-SSIM, the better the fusion performance.

[0115] Table 1 Evaluation results of different methods

[0116]

[0117]

[0118] Specifically, Table 1 lists in detail the comparative data of the method of this application with several other algorithms on various evaluation indicators, thus comprehensively demonstrating the performance and advantages of the method of this application. The method of this application has achieved significant improvements in various indicators, which shows that the method of this paper can guarantee the quality of the fused image. Combining visual evaluation and objective indicator evaluation, the infrared and visible light image fusion method for complex lighting environments given in this embodiment can integrate detailed information from two source images of different modalities and produce a fusion result with rich details.

[0119] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention. The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the protection scope of the present invention.

Claims

1. A method for infrared and visible light image fusion under complex lighting environments, characterized in that, Includes the following steps: S1: Obtain multiple pairs of registered infrared and visible light images under various scenarios to form a training image subset, thereby obtaining a training dataset, and then use a scene-aware network to label the training dataset; S2: Input a subset of training images identified as nighttime scenes into the mask modeling algorithm to generate a mask to guide network training; S3: Construct a hierarchical infrared-visible image fusion network; S4: Construct global and local intensity loss functions based on scene-aware networks and mask modeling algorithms to guide the training of hierarchical infrared-visible image fusion networks; The hierarchical infrared-visible image fusion network includes: a shallow feature fusion layer, a mid-level gradient enhancement layer, and a deep semantic interaction layer; the shallow feature fusion layer includes a feature reuse extraction module. The intermediate gradient enhancement layer includes a gradient enhancement module; the deep semantic interaction layer includes a multi-scale semantic interaction module. The hierarchical infrared-visible image fusion network is constructed as follows: Infrared and visible light images from the training dataset are input into the encoder of the network model to extract features. Features from layer i are extracted separately. The features of the infrared and visible light images in the shallow feature fusion layer are concatenated along the channel dimension and then input into the feature reuse extraction module for further feature extraction and reuse. The gradient enhancement module in the middle gradient enhancement layer is used to enhance gradient texture features at the level of mixed features, concatenating infrared and visible light image features in the channel dimension; The multi-scale semantic interaction module based on multi-scale cross-attention in the deep semantic interaction layer is used to integrate deep features; Each layer of features is input into the decoder and concatenated with the features of the previous layer in the channel dimension to restore the fused image.

2. The infrared-visible light image fusion method for complex lighting environments according to claim 1, characterized in that: In S1, the scene-aware network includes: a convolutional layer, a batch normalization layer, a ReLU activation function, a max pooling layer, and a fully connected layer; The training dataset is identified using a scene-aware network to output the probability that the training dataset belongs to a daytime scene or a nighttime scene.

3. The infrared-visible light image fusion method for complex lighting environments according to claim 1, characterized in that: In S2, the mask modeling algorithm is as follows: the visible light images in the training image subset are input into the scene perception network for identification; the images identified as night scenes are subjected to Gaussian filtering; the Gaussian filtering result is binarized; the binarized result is processed; and the processed binarized result is superimposed with the person semantic segmentation label to generate a mask to guide the network training.

4. The infrared-visible light image fusion method for complex lighting environments according to claim 1, characterized in that: The gradient enhancement module designed in the middle layer specifically refers to: The mixed features of the input gradient enhancement module are first passed through a convolutional layer. Then, the Sobel operator, Laplacian operator, and Prewitt operator are used to calculate the gradients of the features output by the first convolutional layer. The gradient features calculated by the three gradient operators are then concatenated along the channel dimension and fed into the convolutional layer. Finally, the mixed features of the initial input are concatenated along the channel dimension and fed into the convolutional layer.

5. The infrared-visible light image fusion method for complex lighting environments according to claim 1, characterized in that: The multi-scale semantic interaction module is: We use three convolutional layers with different kernel sizes to capture multi-scale deep features. Then, we concatenate the features at different scales along the channel dimension and deploy a projection function that includes a convolutional kernel reshaping operation to convert the multi-scale features into keys and values. We calculate the modality attention map corresponding to each modality and multiply the value by the corresponding attention to obtain features with global context. Then, we add a reshaping operation to the global features and use 1×1 convolution to adjust the channels before concatenating them along the channel dimension to generate fusion features rich in semantics.

6. The infrared-visible light image fusion method for complex lighting environments according to claim 1, characterized in that: The method for constructing a loss function to guide the training of a hierarchical infrared and visible light image fusion network is as follows: a weighted combination of global fusion loss and local information-supplemented fusion loss is used to obtain the total loss of the fusion network.

7. The infrared-visible light image fusion method for complex lighting environments according to claim 6, characterized in that: The global fusion loss consists of: intensity loss, texture loss, and style loss. In the case of daytime, where the light source is singular, using the intensity loss with the highest pixel value is reasonable for extracting significant regions from both modalities. For nighttime conditions, due to localized exposure issues, a mask generated by a mask modeling algorithm is used to construct the intensity loss at night. The scene probability output by the scene-aware network is used to guide the network to avoid localized light source exposure areas in the visible light image, while retaining salient targets and textures within the area from the infrared mode.

8. The infrared-visible light image fusion method for complex lighting environments according to claim 6, characterized in that: The loss from the local information supplementation and fusion is: Considering the severe information loss in local exposure areas of visible light images, local information is used to supplement the fusion loss to guide the hierarchical infrared-visible light image fusion network to integrate information from infrared modes into the fused image.

Images