A method and system for infrared and visible light fusion in mixed degraded environments

Abstract

A method and system for fusion of infrared and visible light in mixed degradation environments includes: 1. constructing a degraded image library and a training dataset; 2. inputting the preprocessed training dataset and the degraded image library into a VGG convolutional neural network to obtain five-level feature maps for each degradation scenario; 3. using a multi-scale degradation estimation module to evaluate the similarity between a source image and multiple degraded images with different degradation scenarios to obtain a degradation factor for each scenario; 4. inputting visible light and infrared images into an image fusion module to obtain a fused image; 5. constructing a loss function, calculating the total loss, and repeating steps 2 to 5 until the function converges, whereupon the weights of the image fusion module are updated; and 6. testing the trained image fusion module. This method can adaptively handle a variety of degradation types and degrees, eliminating reliance on a single predefined degradation scenario, thereby improving the model's applicability in complex real-world scenarios.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an infrared and visible light fusion method and system for a mixed degraded environment. Background Art

[0002] Due to hardware limitations, a single type of sensor cannot fully capture all the details of the imaging scene. For example, infrared sensors can capture thermal radiation and effectively identify salient targets such as pedestrians and vehicles, but lack detailed descriptions of the scene. In contrast, visible light images typically contain rich texture and detail information but are prone to losing objects in the scene due to occlusion. In addition, sensors configured with a single shooting setting also have limited information acquisition in the imaging scene. For example, a camera lens is limited by its depth of field and cannot focus on all objects simultaneously. Image fusion technology combines multi-source images from different sensors or shooting configurations into a high-quality fused image, enriching the scene description and enhancing visual perception. Image fusion technology has a wide range of applications in various fields, including civil, industrial, and security.

[0003] In recent years, deep learning has made significant progress in the field of image fusion. Deep learning-based methods utilize unique network branches to achieve differentiated feature extraction. Furthermore, these methods can learn adaptive fusion strategies. Although deep learning-based methods are effective in integrating raw image information, several challenges remain. Most methods are designed based on standard scenes with clear visual clarity, thus ignoring the difficulties that mixed degradation (such as snow, fog, and rain) present in real-world scenes poses to image fusion.

[0004] While some methods have been specifically designed to address the challenges of image fusion under specific degradation conditions, these methods typically rely on predefined degradation conditions and are primarily applicable to simple degraded scenarios. However, it is worth noting that extreme real-world environments often involve multiple degradation factors, such as snow, fog, and rain, which can degrade the quality of source images and affect fusion performance. It can be observed that real-world images may be affected not only by a single degradation but also by multiple mixed degradations simultaneously. Therefore, it is necessary to develop image fusion methods that consider mixed degradations. Summary of the Invention

[0005] The present invention provides an infrared and visible light fusion method and system for mixed degraded environments to solve the technical problems mentioned in the background technology.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] The present invention provides a method for fusion of infrared and visible light in a mixed degraded environment, comprising the following steps:

[0008] S1. Build an unpaired degraded image library and a training dataset, and preprocess the visible light images in the training dataset;

[0009] S2. Select a source image from the preprocessed training dataset, then select one degraded image with different degradation conditions from the unpaired degraded image library, and input the source image and multiple degraded images with different degradation conditions into the VGG convolutional neural network for feature extraction, obtaining five hierarchical feature maps for each degradation condition;

[0010] S3, the multi-scale degradation estimation module evaluates the similarity between the source image and multiple degraded images with different degradation conditions based on the five-level feature maps of each degradation condition, and obtains the degradation factor of each different degradation condition;

[0011] S4, inputting the visible light image and the infrared image in the source image selected in S2 into an image fusion module to obtain a fused image;

[0012] S5. Construct a contrast fusion total loss function based on the degradation factors of each degradation situation, calculate the total loss of each part between the source image and the fused image, and repeat S2 to S5 until the contrast fusion total loss function converges. Then, use the backpropagation algorithm to update the weights of the image fusion module to obtain the trained image fusion module.

[0013] S6. Test the trained image fusion module to obtain a new fused image.

[0014] Furthermore, the S1 specifically includes the following steps:

[0015] S11. Collect degraded images I containing three degradation conditions: rain, snow, and fog from the image restoration field dataset - , collect A images of each degradation condition to obtain an unpaired degradation image library;

[0016] S12. Collect positive sample images I under normal lighting conditions from the image restoration dataset. + A total of B images are obtained, and multiple source images I are obtained. The source image I includes a visible light image I a and visible light image I a The corresponding infrared image I b ; and construct a training dataset using multiple source images I;

[0017] S13, multiple visible light images I in the training data set a Generate multiple fog images with consistent depth through the atmospheric diffusion model, and then use multiple visible light images I in the training dataset to generate aMultiple rain scene images and snow scene images are obtained through additive composite model simulation respectively;

[0018] S14, then performing multi-scale transformation on the multiple foggy scene images, the multiple rainy scene images, and the multiple snowy scene images so that the size of each image is consistent; thus, a preprocessed training data set is obtained.

[0019] Furthermore, the S2 specifically includes the following steps:

[0020] S21, first select a source image I from the preprocessed training data set, and convert the visible light image I in the source image I into a take out;

[0021] S22. Select a degraded image of the i-th degradation condition from the unpaired degraded image library Where i≤N, N represents the total number of different degradation cases;

[0022] S23, the visible light image I obtained in S21 a The degraded image of the i-th degradation case selected in S22 is input into the VGG convolutional neural network, and the feature map is extracted through the convolution layer 1 and the convolution layer 2 to obtain the feature Figure 1 , then the feature Figure 1 After processing by the maximum pooling layer 1, the first hierarchical feature map C1 is obtained;

[0023] S24, through the convolution layer 3 and the convolution layer 4, the first level feature map C1 is extracted to obtain the feature Figure 2 , and the features Figure 2 Input into the second maximum pooling layer to obtain the second level feature map C2;

[0024] S25, through the convolution layer 5, convolution layer 6 and convolution layer 7, the second level feature map C2 is extracted to obtain the feature Figure 3 , and the features Figure 3 Input into the third max pooling layer to obtain the third level feature map C3;

[0025] S26, performing feature extraction on the third level feature map C3 through convolutional layers 8, 9, and 10 to obtain a feature map 4, and inputting the feature map 4 into the maximum pooling layer 4 to obtain a fourth level feature map C4;

[0026] S27, extracting features from the fourth hierarchical feature map C4 through the convolutional layer 11, the convolutional layer 12, and the convolutional layer 13 to obtain a feature map C5, and sequentially inputting the feature map C5 into the maximum pooling layer 5, three fully connected layers, and a normalization layer to obtain a fifth hierarchical feature map C5; thus, five hierarchical feature maps of the i-th degradation case are obtained;

[0027] S28. Loop S22 to S27 until five-level feature maps of all N degradation conditions are obtained.

[0028] Furthermore, the S3 specifically includes the following steps:

[0029] S31, the multi-scale degradation estimation module evaluates the similarity between the source image and the degraded image of the i-th degradation case based on the five hierarchical feature maps of the i-th degradation case, and obtains the degradation factor of the i-th degradation case;

[0030] S32. Repeat S31 multiple times until the degradation factor of each different degradation situation is obtained.

[0031] Furthermore, the calculation formula of the degradation factor in S31 is as follows:

[0032]

[0033] Among them, λ i Represents the degradation factor of the i-th degradation case, i∈[1,N]; j represents the j-th level feature map; represents the Frobenius norm; d i It is the dimension of query value and key value; represents the transpose of the key vector of the j-th level feature map; T represents the transpose of the matrix; Q ij represents the query vector composed of the j-th level feature map of the degraded image of the i-th degradation case; where the query vector Q ij Sum value vector V j The following relationship is satisfied:

[0034]

[0035] K j =V j =W k ·C j (I) = W v ·C j (I)

[0036] Among them, W q The weight matrix used to generate the query vector; K j W represents the key vector of the j-th level feature map; k Represents the weight matrix used to generate the key vector; W v represents the weight matrix used to generate the value vector; represents the degraded image of the i-th degradation case.

[0037] Furthermore, the step of constructing the comparison fusion total loss function by combining the degradation factors of each different degradation situation in S5 specifically includes the following steps:

[0038] S51. Construct a unified contrastive learning loss function L based on the degradation factor of each different degradation situation C ;

[0039] S52, constructing an infrared image I within the source image I b and visible light image I a The intensity loss function L int , to make the fused image contain more texture information;

[0040] S53. Construct gradient-preserving loss function L gra , to make the fused image contain more gradient information;

[0041] S54, according to the set weighted combination of the unified contrast learning loss function L C , strength loss function L int And the gradient preserving loss function L gra Add them together to get the total loss function of contrast fusion. The specific total loss function of contrast fusion is as follows:

[0042] L=L C +ω1·L gra +ω2·L int

[0043] Among them, L represents the total loss function of contrast fusion, ω1 and ω2 represent the gradient preservation loss function L respectively. gra , strength loss function L int The weight of .

[0044] Furthermore, the unified contrastive learning loss function L in S51 is C The formula is as follows:

[0045]

[0046] Among them, Z represents an index, that is, index one, which represents a cycle between 1 and N; λ z represents the degradation factor associated with index z; Denotes the contrastive learning loss function for the i-th degradation case: The following relationship is satisfied:

[0047]

[0048] Where G(.) represents the Gram matrix, which is used as a quantitative description of the potential image features; d(.) represents the distance between the images at the feature level; I f represents the fused image; represents the positive sample image for the i-th degradation case; αi is a hyperparameter;

[0049] Among them, the Gram matrix G is calculated by the following process:

[0050] S511, first calculate the features Figure 1 To the correlation between different channels in feature map 5 The correlation The calculation process is as follows:

[0051]

[0052] Where, Represents the correlation between the mth channel and the nth channel in the first level feature map: Represents the feature value of the kth spatial position of the mth channel in the first level feature map; Represents the eigenvalue of the kth position of the nth channel in the first layer feature map; k represents the spatial position of each channel; m represents the index for calculating the correlation, that is, index two, and n represents the index for calculating the correlation, that is, index three:

[0053] S512, perform matrix calculation on the hierarchical feature maps of the first five layers and concatenate them along the feature dimension to obtain the Gram matrix G = [G 1 , G 2 ,...,G 5 ].

[0054] Furthermore, the intensity loss function L in S52 int The details are as follows:

[0055]

[0056] Where H and W represent the height and width of the input source image, and |·|1 represents the L1 norm.

[0057] Furthermore, the gradient preservation loss function L in S53 gra The details are as follows:

[0058]

[0059] in, represents the gradient operator.

[0060] A second aspect of the present invention further provides an infrared and visible light fusion system for mixed degraded environments, comprising a computer device programmed or configured to execute the above-mentioned infrared and visible light fusion method;

[0061] Alternatively, a computer program programmed or configured to execute the above-mentioned infrared and visible light fusion method is stored in the memory of the computer device.

[0062] Beneficial effects of the present invention:

[0063] 1. This paper proposes an image fusion module for mixed degradation environments, which can adaptively handle a variety of degradation conditions of different types and degrees. It no longer relies on a single predefined degradation scenario, greatly improving the applicability of the image fusion module in complex real-world scenarios.

[0064] 2. The present invention constructs a multi-scale degradation estimation module, which accurately estimates the image degradation factor by performing cross-attention analysis on unpaired images under different interference conditions (i.e., different degradation situations), thereby better integrating the key information in the source images and improving the quality of the fused image.

[0065] 3. The present invention designs a unified contrast fusion loss function, which can maintain high-resolution clear information during the fusion process while staying away from various degraded information, ensuring the clarity and detail retention of the fused image in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is the overall flow chart of the infrared and visible light fusion method of the present invention;

[0067] Figure 2 This is a schematic diagram of the network structure of the multi-scale degradation estimation module in the present invention:

[0068] Figure 3 This is a structural diagram of the VGG convolutional neural network in the present invention. DETAILED DESCRIPTION

[0069] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present invention. However, the present invention may be implemented in many other forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present disclosure.

[0070] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0072] Reference Figure 1 The embodiment of the present application provides a method for fusion of infrared and visible light in a mixed degraded environment, comprising the following steps:

[0073] S1. Build an unpaired degraded image library and a training dataset, and preprocess the visible light images in the training dataset;

[0074] S2. Select a source image from the preprocessed training dataset, and then select one degraded image with different degradation conditions from the unpaired degraded image library. Input the source image and multiple degraded images with different degradation conditions into the VGG convolutional neural network for feature extraction, and obtain five hierarchical feature maps for each degradation condition:

[0075] S3, the multi-scale degradation estimation module evaluates the similarity between the source image and multiple degraded images with different degradation conditions based on the five-level feature maps of each degradation condition, and obtains the degradation factor of each different degradation condition;

[0076] S4, inputting the visible light image and the infrared image in the source image selected in S2 into an image fusion module to obtain a fused image;

[0077] S5. Construct a contrast fusion total loss function based on the degradation factors of each degradation situation, calculate the total loss of each part between the source image and the fused image, and repeat S2 to S5 until the contrast fusion total loss function converges. Then, use the backpropagation algorithm to update the weights of the image fusion module to obtain the trained image fusion module.

[0078] S6. Test the trained image fusion module to obtain a new fused image.

[0079] The present invention proposes an image fusion module for mixed degradation environments, which can adaptively handle multiple types and degrees of degradation and no longer relies on a predefined single degradation scenario, greatly improving the applicability of the image fusion module in complex real-world scenarios.

[0080] In some embodiments, the step S1 specifically includes the following steps:

[0081] S11. Collect degraded images I containing three degradation conditions: rain, snow, and fog from the image restoration field dataset -, 200 images of each degradation condition are collected to obtain the unpaired degraded image library;

[0082] S12. Collect positive sample images I under normal lighting conditions from the image restoration dataset. + A total of 100, or other number of images, are obtained from the image fusion dataset. The source image I includes the visible light image I a and visible light image I a The corresponding infrared image I b : and construct a training dataset using multiple source images I;

[0083] S13, multiple visible light images I in the training data set a Generate multiple fog images with consistent depth through the atmospheric diffusion model, and then use multiple visible light images I in the training dataset to generate a Multiple rain scene images and snow scene images are obtained through additive composite model simulation respectively;

[0084] S14, then performing multi-scale transformation on the multiple foggy scene images, the multiple rainy scene images, and the multiple snowy scene images so that the size of each image is 224×224; thus, a preprocessed training data set is obtained.

[0085] In some embodiments, reference Figure 1 , the S2 specifically includes the following steps:

[0086] S21, first select a source image I from the preprocessed training data set, and convert the visible light image I in the source image I into a take out;

[0087] S22. Select a degraded image of the i-th degradation condition from the unpaired degraded image library Where i≤N, N represents the total number of different degradation cases; preferably, the value of N is 3;

[0088] S23, such as Figure 3 As shown, the visible light image I obtained by S21 a The degraded image of the i-th degradation case selected in S22 is input into the VGG convolutional neural network, and the feature map is extracted through the convolution layer 1 and the convolution layer 2 to obtain the feature Figure 1 , then the feature Figure 1 After processing by the maximum pooling layer 1, the first hierarchical feature map C1 is obtained: the number of output channels of the convolution layer 1 and the convolution layer 2 is 64, the convolution kernel size is 3*3, and the stride is set to 1. Then, after the maximum pooling layer 1, the image size becomes half of the original size; that is, the first hierarchical feature map C1 is half of the original size:

[0089] S24, through the convolution layer 3 and the convolution layer 4, the first level feature map C1 is extracted to obtain the feature Figure 2 , and the features Figure 2 The input is sent to the second max pooling layer to obtain the second hierarchical feature map C2; the number of output channels of convolutional layers 3 and 4 is 128, the convolution kernel size is 3*3, and the stride is set to 1. Then, after the max pooling layer 2, the image size becomes half of the output size of the previous max pooling layer 1;

[0090] S25, through the convolution layer 5, convolution layer 6 and convolution layer 7, the second level feature map C2 is extracted to obtain the feature Figure 3 , and the features Figure 3 The input is sent to the third max pooling layer to obtain the third hierarchical feature map C3; the number of output channels of the fifth, sixth and seventh convolutional layers is 256, the convolution kernel size is 3*3, and the stride is set to 1. After passing through one max pooling layer, the image size becomes half of the output size of the previous max pooling layer 2;

[0091] S26, feature extraction is performed on the third level feature map C3 through convolutional layers eight, nine, and ten to obtain feature map four, and feature map four is input into the maximum pooling layer four to obtain the fourth level feature map C4; the number of output channels of convolutional layers eight, nine, and ten is 256, the convolution kernel size is 3*3, and the stride is set to 1. After passing through one layer of maximum pooling layer, the image size becomes half of the output size of the previous maximum pooling layer three;

[0092] S27, extracting features from the fourth hierarchical feature map C4 through the convolutional layer 11, the convolutional layer 12, and the convolutional layer 13 to obtain a feature map C5, and sequentially inputting the feature map C5 into the maximum pooling layer 5, three fully connected layers, and a normalization layer to obtain a fifth hierarchical feature map C5; thus, five hierarchical feature maps of the i-th degradation case are obtained;

[0093] The number of output channels of convolutional layers 11, 12, and 13 is 256, the convolution kernel size is 3*3, and the stride is set to 1. After passing through one max pooling layer, the image size becomes half of the output size of the previous max pooling layer 4.

[0094] The first and second level feature maps C1 and C2 are shallow features, covering details such as texture and shape. The third, fourth, and fifth level feature maps C3, C4, and C5 are deeper feature maps, mainly containing content or spatial structure. This combination of shallow and deep features forms a comprehensive information set that is not easily perceived by the human visual system.

[0095] S28. Loop S22 to S27 until five-level feature maps of all N degradation conditions are obtained.

[0096] In some embodiments, S3 specifically includes the following steps:

[0097] S31, the multi-scale degradation estimation module evaluates the similarity between the source image and the degraded image of the i-th degradation situation based on the five hierarchical feature maps of the i-th degradation situation; obtains the degradation factor of the i-th degradation situation; such as Figure 2 As shown in Figure 2, applying cross-attention at the feature level provides significant advantages for image similarity assessment. By dynamically modeling the correlation between features of two images, a more informative and global feature representation can be achieved:

[0098] S32. Repeat S31 multiple times, computing the cross-attention between the feature maps of the visible light image and the N degraded images layer by layer. Then, the Frobenius norm is calculated and normalized to obtain degradation factors representing the N different degradation scenarios. A higher degradation factor indicates a greater similarity between the source image I and the degradation scenario, and therefore requires more processing to eliminate that particular type of degradation.

[0099] In some embodiments, the calculation formula of the degradation factor in S31 is as follows:

[0100]

[0101] Among them, λ i Represents the degradation factor of the i-th degradation case, i∈[1,N]: j represents the j-th level feature map; represents the Frobenius norm: d i It is the dimension of query value and key value; Represents the transpose of the key vector of the j-th level feature map: T represents the transpose of the matrix; Q ij represents the query vector composed of the j-th level feature map of the degraded image of the i-th degradation case; where the query vector Q ij Sum value vector V j The following relationship is satisfied:

[0102]

[0103] K j =V j =W k ·C j (I) = W v ·C j (I)

[0104] Among them, W q The weight matrix used to generate the query vector; Kj Represents the key vector of the j-th level feature map: W k Represents the weight matrix used to generate the key vector: W v Represents the weight matrix used to generate the value vector: represents the degraded image of the i-th degradation case.

[0105] The present invention constructs a multi-scale degradation estimation module, which accurately estimates the image degradation factor by performing cross-attention analysis on unpaired images under different interference conditions (i.e., different degradation situations), thereby better fusing the key information in the source images and improving the quality of the fused image.

[0106] In some embodiments, the step of constructing a comparative fusion total loss function by combining the degradation factors of each different degradation situation in S5 specifically includes the following steps:

[0107] S51. Construct a unified contrastive learning loss function L based on the degradation factor of each different degradation situation C ;

[0108] S52, constructing an infrared image I within the source image I b and visible light image I a The intensity loss function L int , to make the fused image contain more texture information;

[0109] S53. Construct gradient-preserving loss function L gra , to make the fused image contain more gradient information;

[0110] S54, according to the set weighted combination of the unified contrast learning loss function L C , strength loss function L int And the gradient preserving loss function L gra Add them together to get the total loss function of contrast fusion. The specific total loss function of contrast fusion is as follows:

[0111] L=L C +ω1·L gra +ω2·L int

[0112] Among them, L represents the total loss function of contrast fusion, ω1 and ω2 represent the gradient preservation loss function L respectively. gra , strength loss function L int The weight of .

[0113] This paper designs a unified contrast fusion loss function that maintains high-resolution, clear information during the fusion process while avoiding various degraded information, ensuring the clarity and detail preservation of the fused image in complex scenes. Furthermore, the proposed contrast fusion loss function can handle image fusion problems in a variety of complex, degraded scenarios without relying on predefined degradation conditions, significantly improving the quality and applicability of the fused image.

[0114] In some embodiments, the unified contrastive learning loss function L in S51 is C The formula is as follows:

[0115]

[0116] Among them, Z represents an index, that is, index one, which represents a cycle between 1 and N; λ z represents the degradation factor associated with index z; represents the contrastive learning loss function for the i-th degradation case; The following relationship is satisfied:

[0117]

[0118] Where G(.) represents the Gram matrix, which is used as a quantitative description of the potential image features; d(.) represents the distance between images at the feature level, and the closer the distance, the more similar the images are; I f represents the fused image; represents the positive sample image for the i-th degradation case; α i is a hyperparameter;

[0119] Among them, the Gram matrix G is calculated by the following process:

[0120] S511, first calculate the features Figure 1 To the correlation between different channels in feature map 5 The correlation The calculation process is as follows:

[0121]

[0122] Where, Represents the correlation between the mth channel and the nth channel in the first level feature map: Represents the feature value of the kth spatial position of the mth channel in the first level feature map; Represents the eigenvalue of the kth position of the nth channel in the first layer feature map: k represents the spatial position of each channel; m represents the index for calculating the correlation, that is, index two, and n represents the index for calculating the correlation, that is, index three:

[0123] S512, perform matrix calculation on the hierarchical feature maps of the first five layers and concatenate them along the feature dimension to obtain the Gram matrix G = [G 1 , G 2 ,...,G 5 ].

[0124] In some embodiments, the intensity loss function L in S52 is int The details are as follows:

[0125]

[0126] Where H and W represent the height and width of the input source image, and |·|1 represents the L1 norm.

[0127] In some embodiments, the gradient preservation loss function L in S53 gra The details are as follows:

[0128]

[0129] in, represents the gradient operator.

[0130] A second aspect of the present invention further provides an infrared and visible light fusion system for a mixed degraded environment, comprising a computer device programmed or configured to execute the above-mentioned infrared and visible light fusion method:

[0131] Alternatively, a computer program programmed or configured to execute the above-mentioned infrared and visible light fusion method is stored in the memory of the computer device.

[0132] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in this field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope required by the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for fusion of infrared and visible light in mixed degraded environments, characterized in that: The steps include: S1. Build an unpaired degraded image library and a training dataset, and preprocess the visible light images in the training dataset; S2. Select a source image from the preprocessed training dataset, then select one degraded image with different degradation conditions from the unpaired degraded image library, and input the source image and multiple degraded images with different degradation conditions into the VGG convolutional neural network for feature extraction, obtaining five hierarchical feature maps for each degradation condition; S3, the multi-scale degradation estimation module evaluates the similarity between the source image and multiple degraded images with different degradation conditions based on the five-level feature maps of each degradation condition, and obtains the degradation factor of each different degradation condition; S4, inputting the visible light image and the infrared image in the source image selected in S2 into an image fusion module to obtain a fused image; S5. Construct a unified contrastive learning loss function L based on the degradation factors of each different degradation situation C , then based on the unified contrastive learning loss function L C Construct a contrast fusion total loss function, calculate the total loss of each part between the source image and the fused image, loop S2 to S5 until the contrast fusion total loss function converges, and then use the back propagation algorithm to update the weights of the image fusion module to obtain the trained image fusion module; The unified contrastive learning loss function L C The formula is as follows: Where Z represents an index, i.e., index 1, which represents a cycle between 1 and N, and N represents the total number of different degradation cases; λ i represents the degradation factor of the i-th degradation case; λ z represents the degradation factor associated with index z; represents the contrastive learning loss function for the i-th degradation case; The following relationship is satisfied: Where G(.) represents the Gram matrix, which is used as a quantitative description of the potential image features; d(.) represents the distance between the images at the feature level; I f represents the fused image; represents the positive sample image for the i-th degradation case; α i is a hyperparameter; represents the degraded image of the i-th degradation case; S6. Testing the trained image fusion module to obtain a new fused image; The S3 specifically includes the following steps: S31, the multi-scale degradation estimation module evaluates the similarity between the source image and the degraded image of the i-th degradation case based on the five hierarchical feature maps of the i-th degradation case, and obtains the degradation factor of the i-th degradation case; S32. Repeat S31 multiple times to calculate the cross attention between the feature maps of the visible light image and the degraded images of N degradation situations layer by layer, then calculate the Frobenius norm and normalize the result to obtain the degradation factors representing N different degradation scenarios.

2. The infrared and visible light fusion method according to claim 1, characterized in that: The S1 specifically includes the following steps: S11. Collect degraded images I containing three degradation conditions: rain, snow, and fog from the image restoration field dataset - , collect A images of each degradation condition to obtain an unpaired degradation image library; S12. Collect positive sample images I under normal lighting conditions from the image restoration dataset. + A total of B images are obtained, and multiple source images I are obtained. The source image I includes a visible light image I a and visible light image I a The corresponding infrared image I b ; And construct a training dataset using multiple source images I; S13, multiple visible light images I in the training data set a Generate multiple fog images with consistent depth through the atmospheric diffusion model, and then use multiple visible light images I in the training dataset to generate a Multiple rain scene images and snow scene images are obtained through additive composite model simulation respectively; S14, then performing a multi-scale transformation on the multiple fog scene images, the multiple rain scene images, and the multiple snow scene images so that the size of each image is consistent; At this point, the preprocessed training dataset is obtained.

3. The infrared and visible light fusion method according to claim 2, characterized in that: The S2 specifically includes the following steps: S21, first select a source image I from the preprocessed training data set, and convert the visible light image I in the source image I into a take out; S22. Select a degraded image of the i-th degradation condition from the unpaired degraded image library Where i≤N, N represents the total number of different degradation cases; S23, the visible light image I obtained in S21 a The degraded image of the i-th degradation case selected in S22 is input into the VGG convolutional neural network, and feature maps are extracted through convolutional layers 1 and 2 to obtain feature map 1. Then, feature map 1 is processed through the maximum pooling layer 1 to obtain the first hierarchical feature map C1. S24, performing feature extraction on the first level feature map C1 through convolutional layers 3 and 4 to obtain feature map 2, and inputting feature map 2 into the maximum pooling layer 2 to obtain the second level feature map C2; S25, performing feature extraction on the second level feature map C2 through convolutional layers 5, 6, and 7 to obtain feature map 3, and inputting feature map 3 into the maximum pooling layer 3 to obtain the third level feature map C3; S26, performing feature extraction on the third level feature map C3 through convolutional layers 8, 9, and 10 to obtain a feature map 4, and inputting the feature map 4 into the maximum pooling layer 4 to obtain a fourth level feature map C4; S27, extracting features from the fourth hierarchical feature map C4 through the convolutional layer 11, the convolutional layer 12, and the convolutional layer 13 to obtain a feature map C5, and sequentially inputting the feature map C5 into the maximum pooling layer 5, three fully connected layers, and a normalization layer to obtain a fifth hierarchical feature map C5; thus, five hierarchical feature maps of the i-th degradation case are obtained; S28. Loop S22 to S27 until five-level feature maps of all N degradation conditions are obtained.

4. The infrared and visible light fusion method according to claim 3, characterized in that: The calculation formula of the degradation factor in S31 is as follows: Among them, j represents the j-th level feature map; represents the Frobenius norm; d i It is the dimension of query value and key value; represents the transpose of the key vector of the j-th level feature map; T represents the transpose of the matrix; Q ij represents the query vector composed of the j-th level feature map of the degraded image of the i-th degradation case; where the query vector Q ij Sum value vector V j The following relationship is satisfied: K j =V j =W k ·C j (I)=W v ·C j (I) Among them, W q The weight matrix used to generate the query vector; K j W represents the key vector of the j-th level feature map; k Represents the weight matrix used to generate the key vector; W v represents the weight matrix used to generate the value vector; represents the degraded image of the i-th degradation case.

5. The infrared and visible light fusion method according to claim 4, characterized in that: The step of constructing the comparison fusion total loss function by combining the degradation factors of each different degradation situation in S5 specifically includes the following steps: S51. Construct a unified contrastive learning loss function L based on the degradation factor of each different degradation situation C ; S52, constructing an infrared image I within the source image I b and visible light image I a The intensity loss function L int , to make the fused image contain more texture information; S53. Construct gradient-preserving loss function L gra , to make the fused image contain more gradient information; S54, according to the set weighted combination of the unified contrast learning loss function L C , strength loss function L int And the gradient preserving loss function L gra Add them together to get the total loss function of contrast fusion. The specific total loss function of contrast fusion is as follows: L=L C +ω1·L gra +ω2·L int Among them, L represents the total loss function of contrast fusion, ω1 and ω2 represent the gradient preservation loss function L respectively. gra , strength loss function L int The weight of .

6. The infrared and visible light fusion method according to claim 5, characterized in that: The Gram matrix G is calculated by the following process: S511, first calculate the correlation between different channels in feature map 1 to feature map 5 The correlation The calculation process is as follows: Where, Represents the correlation between the mth channel and the nth channel in the first level feature map; Represents the feature value of the kth spatial position of the mth channel in the first level feature map; Represents the feature value of the k-th position of the n-th channel in the first layer feature map; k represents the spatial position of each channel; m represents the index for calculating the correlation, that is, index 2, and n represents the index for calculating the correlation, that is, index 3; S512, perform matrix calculation on the hierarchical feature maps of the first five layers and concatenate them along the feature dimension to obtain the Gram matrix G = [G 1 , G 2 ,...,G 5 ].

7. The infrared and visible light fusion method according to claim 6, characterized in that: The intensity loss function L in S52 int The details are as follows: Where H and W represent the height and width of the input source image, and |·|1 represents the L1 norm.

8. The infrared and visible light fusion method according to claim 7, characterized in that: The gradient preservation loss function L in S53 gra The details are as follows: in, represents the gradient operator.

9. An infrared and visible light fusion system for mixed degraded environments, including computer equipment, characterized in that: The computer device is programmed or configured to perform the infrared and visible light fusion method according to any one of claims 1 to 8; Alternatively, a computer program programmed or configured to execute the infrared and visible light fusion method according to any one of claims 1 to 8 is stored in the memory of the computer device.

Images