Infrared and visible light image fusion system based on double-weight supervision

Abstract

The invention provides an infrared and visible light image fusion system based on double-weight supervision. The infrared and visible light image fusion system comprises a generator based on a weight supervision module, a double-weight supervision constraint strategy and a discriminator, the generator based on the weight supervision module is used for generating a fused image according to the infrared image and the visible light image; the double-weight supervised constraint strategy is used for improving feature extraction and fusion performance of the generator; and the discriminator is used for enabling the fused image and the source image to maintain the same information distribution. The fusion image is generated through the generator, a constraint strategy of double-weight supervision is adopted, different modal features of the fusion image and correlation between the different modal features and corresponding modal source image features are utilized, the feature extraction and fusion capacity of the model is improved, negative influences of redundant information on the fusion result are effectively avoided, and therefore the purpose of improving the fusion effect is achieved.

Description

Technical Field

[0001] This invention belongs to the field of digital image processing technology, and in particular relates to an infrared and visible light image fusion system based on dual-weight supervision. Background Technology

[0002] Due to various factors such as environment and technological principles, a single type of sensor cannot comprehensively and effectively describe a scene. Thermal imaging sensors generate infrared images by capturing the thermal radiation emitted by objects. Unaffected by lighting or occlusion, they can effectively highlight prominent targets such as pedestrians and vehicles, but their spatial resolution is low, making it difficult to represent the detailed texture information of the scene. In contrast, visible light images captured by optical sensors contain rich detailed texture information and have high spatial resolution, but are susceptible to interference from factors such as lighting, weather, camouflage, and occlusion. Infrared and visible image fusion (IVIF) technology aims to fuse complementary information from different modal source images while removing redundant information. This technology not only improves the ability of humans and computers to understand scenes but also significantly enhances the performance of advanced vision tasks such as target detection and semantic segmentation, and is therefore widely used in fields such as military reconnaissance, security monitoring, and autonomous driving.

[0003] Over the past few decades, various IVIF methods have been proposed, which can be mainly divided into traditional methods and deep learning-based methods. Traditional IVIF methods typically include five categories: methods based on multi-scale transformation, methods based on subspace, methods based on sparse representation, methods based on saliency representation, and others. While these traditional methods have achieved good results, they have certain limitations due to the hand-designed activity level measurement methods resulting in insufficient extracted image features. Furthermore, when dealing with different and complex scenes, the fusion rules are too complex and computationally expensive.

[0004] With the advancement of computer hardware performance and deep learning technology, a number of deep learning-based IVIF methods have rapidly emerged in this field. Currently, the main methods include those based on Convolutional Neural Networks (CNNs), Autoencoders (AEs), Generative Adversarial Networks (GANs), Transformers, and others. Although deep learning-based IVIF methods have significantly improved fusion performance compared to traditional methods, they still face some pressing challenges.

[0005] First, existing methods tend to transfer as much information from the source image as possible to the fused image, neglecting the potential degradation in fused image quality due to the introduction of redundant information. To fully preserve information from the source image in the fusion result, Zhang et al. proposed SDNet, a pixel-level supervised approach. It uses a compression network to generate the fused image and a decomposition network to decompose the fused image into images identical to the source images. While the pixel-level supervision strategy can retain more information from the source image, redundant information from the source image is also transferred to the fusion result, leading to a decrease in fusion quality.

[0006] Secondly, existing methods struggle to balance fusion performance, network size, and performance in downstream tasks. To improve the model's feature extraction and long-range dependency capture capabilities, the Transformer architecture is increasingly prevalent in the IVIF field. Furthermore, some methods implement IVIF using the idea of ​​diffusion models. While these methods have improved fusion performance, a key limitation exists: in pursuit of superior fusion performance, these methods often employ overly complex models with excessively high parameter counts, making them difficult to deploy on computationally limited edge devices in practical applications. Moreover, these methods ignore the high computational efficiency requirements of IVIF as a prerequisite for advanced computer vision tasks. Although some early methods had relatively simple models and high computational efficiency, their fusion performance and application results in downstream tasks were less than ideal. Summary of the Invention

[0007] In view of the problems existing in the prior art, the present invention provides an infrared and visible light image fusion system based on dual-weight supervision, which at least partially solves the problem of low fusion effect in the prior art.

[0008] This disclosure provides an infrared and visible light image fusion system based on dual-weight supervision, including: a generator based on a weight supervision module, a dual-weight supervision constraint strategy, and a discriminator;

[0009] The generator based on the weighted supervision module is used to generate a fused image from infrared and visible light images;

[0010] The dual-weight supervision constraint strategy is used to improve the feature extraction and fusion performance of the generator;

[0011] The discriminator is used to ensure that the fused image and the source image maintain the same information distribution.

[0012] Optionally, the generator based on the weight supervision module includes: a weight supervision module based on frequency adaptive attention, a cross-weight strategy, and a multi-scale feature fusion module;

[0013] The frequency-adaptive attention-based weight supervision module is used to convert infrared images and visible light images into weight maps;

[0014] The cross-weight strategy is used to calculate the feature map from the weight map;

[0015] The multi-scale feature fusion module is used to fuse feature maps to obtain a fused image.

[0016] Optionally, the method for converting infrared and visible light images into weighted maps includes:

[0017] The source image is downsampled twice, and then shallow features are obtained based on the results of the two downsampling. The shallow features are then downsampled twice more to obtain deep features.

[0018] The deep features are upsampled twice using transposed convolutions, and the results of the two upsamplings are concatenated with the dimensions of the shallow features to obtain a weighted feature that contains both shallow and deep semantic features. Then, the weighted feature is upsampled twice to obtain the weighted feature.

[0019] A weight map is obtained by adding the source image and the weight features pixel by pixel using skip connections.

[0020] Optionally, the step of calculating the feature map from the weight map includes:

[0021] Feature maps of infrared and visible light images are obtained by calculating based on the weight map and adaptive weight balancing parameters.

[0022] Optionally, the adaptive weight balancing parameters are calculated by a module consisting of convolutional, pooling, and fully connected layers.

[0023] Optionally, the method for fusing the feature maps to obtain a fused image includes:

[0024] The infrared and visible light features are spliced ​​together and then expanded into channels to adjust the weights of features in different frequency ranges.

[0025] Optionally, the dual-weight supervision constraint strategy is used to constrain the frequency-adaptive attention-based weight supervision module to extract information from the source image, and to encourage the multi-scale feature fusion module to retain useful information while suppressing the influence of redundant information on the fusion result.

[0026] Optionally, the discriminator adopts a WGAN-GP structure.

[0027] The loss function of the WGAN-GP structure is constructed based on the distribution of real data, the distribution of generated data, the distribution of interpolation between real and generated data, and the weight coefficients of the gradient penalty term.

[0028] Optionally, the loss function of the discriminator includes fusion loss and adversarial loss;

[0029] The fusion loss includes content loss and weight loss;

[0030] The content loss includes intensity loss, gradient loss, and structure loss;

[0031] The weight loss includes dual-weight monitoring loss and complementary weight loss;

[0032] The adversarial loss includes generator loss, infrared discriminator loss, and visible light discriminator loss.

[0033] Optionally, the content loss is used to prompt the generator to generate a fused image with a similar data distribution to the source image;

[0034] The intensity loss is used to measure the pixel-by-pixel intensity difference between the fusion result and the source image, and the gradient loss is used to drive the fusion network to have the ability to preserve texture detail information;

[0035] A multi-scale structural similarity index is used as the structural loss to preserve the brightness, contrast and structural information in the source image;

[0036] The dual-weight supervision loss is calculated from the weight map generated by the frequency-adaptive attention-based weight supervision module. It is used to encourage the frequency-adaptive attention-based weight supervision module to extract feature weights and to constrain the multi-scale feature fusion module to generate a fused image.

[0037] The complementary loss is used to leverage the complementarity of source images from different modalities to encourage a frequency-adaptive attention-based weighted supervision module to focus on complementary regions between different modalities.

[0038] The infrared and visible light image fusion system based on dual-weight supervision provided by this invention generates fused images through a generator and adopts a dual-weight supervision constraint strategy. It utilizes the different modal features of the fused image and their correlation with the features of the corresponding modal source images to improve the feature extraction and fusion capabilities of the model, effectively avoiding the negative impact of redundant information on the fusion results, thereby achieving the goal of improving the fusion effect. Attached Figure Description

[0039] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0040] Figure 1 This is a schematic diagram of the structure of an infrared and visible light image fusion system based on dual-weight supervision provided in an embodiment of this disclosure;

[0041] Figure 2 A structural diagram of the FADC provided in the embodiments of this disclosure;

[0042] Figure 3 This is a schematic diagram of qualitative comparison on the TNO dataset provided in an embodiment of this disclosure;

[0043] Figure 4 This is a schematic diagram of qualitative comparison on the RoadScene dataset provided in an embodiment of this disclosure;

[0044] Figure 5 This is a schematic diagram of qualitative comparison on the MSRS dataset provided in an embodiment of the present disclosure;

[0045] Figure 6 M provided for the embodiments of this disclosure 3 A schematic diagram of qualitative comparisons on the FD dataset;

[0046] Figure 7 This is a schematic diagram showing the comparison of ablation experiments provided in the embodiments of this disclosure;

[0047] Figure 8 This is a schematic diagram illustrating the qualitative comparison of target detection in the MSRS dataset provided in this embodiment of the disclosure;

[0048] Figure 9 This is a schematic diagram illustrating the qualitative comparison of semantic segmentation of the MSRS dataset provided in this embodiment of the disclosure. Detailed Implementation

[0049] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0050] It should be understood that the following specific examples illustrate the implementation of this disclosure, and those skilled in the art can easily understand other advantages and effects of this disclosure from the content disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. This disclosure can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this disclosure. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0051] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0052] It should also be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this disclosure. The illustrations only show the components related to this disclosure and are not drawn according to the number, shape and size of the components in actual implementation. In actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0053] Furthermore, specific details are provided in the following description to facilitate a thorough understanding of the examples. However, those skilled in the art will understand that the described aspects can be practiced without these specific details.

[0054] WGAN-GP refers to Wasserstein generative adversarial networks with gradient penalty;

[0055] FAF2M (Frequency Adaptive Feature Fusion Module) refers to the frequency adaptive feature fusion module;

[0056] CBRD refers to the basic module consisting of C (convolutional layer), B (batch normalization layer), R (activation function layer), and D (average pooling downsampling layer).

[0057] CTBRU refers to the basic module consisting of CT (transposed convolutional layer), B (batch normalization layer), and R (activation function layer);

[0058] FA2M (Freguency Adaptive Attention Module) refers to a frequency-adaptive attention module.

[0059] FADC (Frequency-Adaptive Dilated Convolution) refers to frequency-adaptive dilated convolution.

[0060] WSM (Weight Supervision Module) refers to the weight supervision module;

[0061] This embodiment discloses an infrared and visible light image fusion system based on dual-weight supervision, including: a generator based on a weight supervision module, a constraint strategy based on dual-weight supervision, and a discriminator;

[0062] The generator based on the weighted supervision module is used to generate a fused image from infrared and visible light images;

[0063] The dual-weight supervision constraint strategy is used to improve the feature extraction and fusion performance of the generator;

[0064] The discriminator is used to ensure that the fused image and the source image maintain the same information distribution.

[0065] Optionally, the generator based on the weight supervision module includes: a weight supervision module based on frequency adaptive attention, a cross-weight strategy, and a multi-scale feature fusion module;

[0066] The frequency-adaptive attention-based weight supervision module is used to convert infrared images and visible light images into weight maps;

[0067] The cross-weight strategy is used to calculate the feature map from the weight map;

[0068] The multi-scale feature fusion module is used to fuse feature maps to obtain a fused image.

[0069] Optionally, the method for converting infrared and visible light images into weighted maps includes:

[0070] The source image is downsampled twice, and then shallow features are obtained based on the results of the two downsampling. The shallow features are then downsampled twice more to obtain deep features.

[0071] The deep features are upsampled twice using transposed convolutions, and the results of the two upsamplings are concatenated with the dimensions of the shallow features to obtain a weighted feature that contains both shallow and deep semantic features. Then, the weighted feature is upsampled twice to obtain the weighted feature.

[0072] A weight map is obtained by adding the source image and the weight features pixel by pixel using skip connections.

[0073] Optionally, the step of calculating the feature map from the weight map includes:

[0074] Feature maps of infrared and visible light images are obtained by calculating based on the weight map and adaptive weight balancing parameters.

[0075] Optionally, the adaptive weight balancing parameters are calculated by a module consisting of convolutional, pooling, and fully connected layers.

[0076] Optionally, the method for fusing the feature maps to obtain a fused image includes:

[0077] The infrared and visible light features are spliced ​​together and then expanded into channels to adjust the weights of features in different frequency ranges.

[0078] Optionally, the dual-weight supervision constraint strategy is used to constrain the frequency-adaptive attention-based weight supervision module to extract information from the source image, and to encourage the multi-scale feature fusion module to retain useful information while suppressing the influence of redundant information on the fusion result.

[0079] Optionally, the discriminator adopts a WGAN-GP structure.

[0080] The loss function of the WGAN-GP structure is constructed based on the distribution of real data, the distribution of generated data, the distribution of interpolation between real and generated data, and the weight coefficients of the gradient penalty term.

[0081] Optionally, the loss function of the discriminator includes fusion loss and adversarial loss;

[0082] The fusion loss includes content loss and weight loss;

[0083] The content loss includes intensity loss, gradient loss, and structure loss;

[0084] The weight loss includes dual-weight monitoring loss and complementary weight loss;

[0085] The adversarial loss includes generator loss, infrared discriminator loss, and visible light discriminator loss.

[0086] Optionally, the content loss is used to prompt the generator to generate a fused image with a similar data distribution to the source image;

[0087] The intensity loss is used to measure the pixel-by-pixel intensity difference between the fusion result and the source image, and the gradient loss is used to drive the fusion network to have the ability to preserve texture detail information;

[0088] A multi-scale structural similarity index is used as the structural loss to preserve the brightness, contrast and structural information in the source image;

[0089] The dual-weight supervision loss is calculated from the weight map generated by the frequency-adaptive attention-based weight supervision module. It is used to encourage the frequency-adaptive attention-based weight supervision module to extract feature weights and to constrain the multi-scale feature fusion module to generate a fused image.

[0090] The complementary loss is used to leverage the complementarity of source images from different modalities to encourage a frequency-adaptive attention-based weighted supervision module to focus on complementary regions between different modalities.

[0091] This embodiment proposes a lightweight IVIF network, DWSFusion, based on dual-weight supervision. It consists of a generator based on a cross-sensing strategy, a dual discriminator with a WGAN-GP structure, and a dual-weight supervision constraint strategy. First, a weight supervision module (WSM) is designed to extract intra-modal features from the source images, employing a cross-sensing strategy to enhance inter-modal feature interaction while reducing the network's parameter count. Then, a feature-level dual-weight supervision strategy is proposed, utilizing the different modal features of the fused image and their correlation with the corresponding modal features of the source images to improve the model's feature extraction and fusion capabilities, effectively avoiding the negative impact of redundant information on the fusion result. Finally, a dual discriminator with a WGAN-GP (Wasserstein GAN with Gradient Penalty) structure and a designed MS-SSIM loss constraint ensure that the fusion result maintains the same information distribution as the source images.

[0092] This embodiment mainly includes three parts: a generator based on the weight supervision module, a constraint strategy based on dual weight supervision, and a discriminator, such as... Figure 1 As shown, the generator is responsible for generating a fused image from the infrared and visible light images. A dual-weight supervised constraint strategy is used to improve the feature extraction and fusion performance of WSM and FAF2M in the generator. The discriminator is responsible for ensuring that the fused image maintains the same information distribution as the source image.

[0093] The generator mainly consists of three parts: a weight supervision module based on frequency adaptive attention, a cross-weight strategy, and a multi-scale feature fusion module.

[0094] (1) WSM: First, given a pair of infrared images and visible light images By inputting each of these into a WSM with the same structure, a weight map of the source image for a specific modality is obtained. This process can be represented as follows:

[0095] (1),

[0096] in, and These are weighted graphs for infrared and visible light images, respectively. This is the weight monitoring module.

[0097] WSM comprises three modules: CBRD, CTBRU, and FA2M (Freguency Adaptive Attention Module). The CBRD module consists of a 3×3 convolutional layer, a BatchNorm layer, a ReLU activation layer, and an AvgPooling layer, used to downsample the input and extract deep semantic features. The CTBRU module consists of a transposed convolutional layer, a BatchNorm layer, and a ReLU activation layer, used to recover a weight map of the same size as the input image from deep semantic features. The FA2M module consists of sequential channel attention and frequency-adaptive spatial attention, designed to improve the WSM's weight allocation for different channels and spaces during downsampling and upsampling. Ordinary spatial attention mechanisms typically use convolutional layers with fixed kernel sizes to encode the spatial information of the input feature map; however, this method struggles to balance the relationship between the receptive field of the convolutional kernel and the extraction of high-frequency features. Small convolutional kernels can better focus on high-frequency details in features, but their smaller receptive field limits their ability to extract global information. Large convolutional kernels, while having a larger receptive field, suffer from insufficient attention to high-frequency information and higher computational cost. Frequency-adaptive spatial attention differs from ordinary spatial attention mechanisms in that it adaptively adjusts the dilation rate of the convolutional kernel based on the high-frequency information content of local regions of the input features. FA2M replaces the convolutional layers with a 3×3 FADC (Fast-Augmented Convolutional Digester), effectively balancing the relationship between receptive field size and high-frequency information attention. Furthermore, compared to methods using large convolutional kernels or multi-scale approaches, it effectively reduces computational complexity. The internal structure of the FADC is as follows... As shown.

[0098] Specifically, the source image After inputting WSM, it undergoes two downsampling processes, followed by FA2M to obtain shallow features. Then, deep features were obtained through two more downsampling operations. This process can be represented as

[0099] (2),

[0100] (3),

[0101] in, Indicates 2x downsampling Second-rate.

[0102] Subsequently, in order to inject shallow features into weighted features, two transposed convolutions are used on the deep features. Perform upsampling twice and combine it with shallow features Dimensional concatenation yields weighted features that contain both shallow and deep semantic features. Then, it is upsampled twice to obtain the weighted features. This process can be represented as

[0103] (4)

[0104] (5)

[0105] in, Indicates 2x upsampling Second-rate, This indicates a dimension concatenation operation.

[0106] Finally, to ensure that the weight map retains both spatial details and semantic information, skip connections are used to add the source image and weight features pixel by pixel to obtain the final weight map. , can be represented as

[0107] (6)

[0108] in, This represents the Sigmoid function. This indicates an element-wise addition operation.

[0109] (2) Cross-weighting strategy:

[0110] The weight map represents the weight distribution of information in the source image. Considering the similarities and complementarities between different modalities of the same scene, a cross-weighting strategy is designed for feature map calculation. The process is as follows:

[0111] (7)

[0112] (8)

[0113] (9)

[0114] in, and These are feature maps of infrared and visible light images, respectively. This represents element-wise multiplication. The adaptive weight balancing parameters are calculated from the source image by a module consisting of convolutional, pooling, and fully connected layers.

[0115] The feature map obtained by using the cross-weighting strategy not only fully preserves the useful information in the source image, but also effectively suppresses the redundant information of the source image based on the weight map of another modality source image.

[0116] (3) FAF2M (Multi-Scale Feature Fusion Module): The FAF2M module consists of two parts: FADC and FA2M. To avoid loss of feature information, this module does not use any pooling operation. First, the infrared and visible light feature dimensions are concatenated and then input into the FADC for channel expansion, thereby enhancing the diversity and complexity of information expression and better fusing features of different modalities. To better preserve the detailed information in the shallow features, a skip connection is used to directly pass the feature map to the subsequent layers. Subsequently, the features are processed by FA2M. By automatically adjusting the weights of features in different frequency ranges, important information in different frequency intervals can be effectively focused on, thereby improving the sensitivity and expressive ability of multimodal information. In this process, FA2M can not only dynamically adjust the degree of attention to infrared and visible light features, but also strengthen the model's understanding of the relationship between features of different frequencies, thereby achieving more accurate information aggregation in the feature fusion process. Finally, the multi-channel feature maps are fused using the FADC to obtain a single-channel fused image. This process can be represented as follows:

[0117] (10)

[0118] in, To merge images, Representative feature fusion module, This represents the operation of splicing channel dimensions.

[0119] Feature-level dual-weight supervision strategy: In the IVIF task, source image pairs typically contain both useful and redundant information. SDNet utilizes a designed decomposition network to decompose the fused image into source images of different modalities, thereby encouraging the network to transfer as much information from the source images as possible to the fused image. However, this method ignores the negative impact of redundant information in the source images on the fusion result. To address this issue, this invention proposes a dual-weight supervision constraint strategy. Specifically, the fused image is input into different modalities of the WSM in the generator to obtain the weight maps of the fused result in different modalities. The loss is then calculated by comparing these weight maps with the corresponding weight maps of the source images. This process can be represented as follows:

[0120] (11)

[0121] in, and These represent the weight maps extracted from the fused image using infrared WSM and visible light WSM, respectively.

[0122] This strategy constrains the WSM to better extract useful information from the source image and encourages FAF2M to retain as much useful information as possible while suppressing the impact of redundant information on the fusion result. Compared with SDNet, this strategy does not require training an extra decomposition network, but directly uses the WSM trained during the fusion process to extract the weight map from the fused image, reducing training costs. Furthermore, compared with SDNet's pixel-level supervision strategy, the proposed feature-level dual-weight supervision constraint strategy can effectively avoid the introduction of redundant information from the source image, effectively improving the quality of the fused image.

[0123] Discriminator: DWSFusion can generate relatively ideal fused images using only the generator. To generate even better fused images, a dual discriminator structure is further introduced to encourage the generator network to better retain useful information from the source images. Traditional GAN ​​structures often struggle to balance the training progress of the generator and discriminator, leading to instability during training. WGAN-GP reduces its dependence on the complex structure of the discriminator by replacing JS divergence with Wasserstein distance and removing the Sigmoid activation function constraint from the discriminator output layer. Compared to WGAN, WGAN-GP uses gradient penalty instead of weight pruning, fully leveraging the discriminator's model performance and making it more stable and effective in promoting generator performance. Its loss function is defined as:

[0124] (12)

[0125] in, It is the output of the discriminator. and These are the real data distribution and the generated data distribution, respectively. This represents the distribution of interpolation between real and generated data. It is the weight coefficient of the gradient penalty term.

[0126] The infrared and visible light discriminators share the same structure, each consisting of three CBRD blocks, a Flatten layer, and a fully connected layer. By inputting the source image and the fused image into their respective discriminators, the model generates two feature representation vectors. These two vectors are then averaged to obtain the discriminator's score for each image. The goal is to maximize the discriminator's score for the source image and minimize its score for the generated fused image, thereby making the generated fused image closer to the real image and possessing higher quality. This dual-discriminator structure effectively drives the generator to produce higher-quality fused images. Specifically, the infrared and visible light discriminators guide the generator's training process by analyzing the score differences between the source and fused images, enabling it to progressively optimize its generation strategy. In this way, the generator not only learns how to effectively fuse infrared and visible light information but also ensures that the final fused image exhibits low forgery accuracy in the discriminator's evaluation, i.e., it is as close to the real image as possible. The synergistic effect of the dual discriminators provides the generator with a strong supervisory signal, driving it to continuously improve the generation results, enhancing the quality of the fused image and the effective combination of multimodal information.

[0127] Loss Function: DWSFusion aims to fully integrate useful information from the source images into the fused image while effectively suppressing the negative impact of redundant information on the fusion result. To achieve this goal, the loss function... Due to fusion loss and combat losses Composed of two parts, it can be defined as:

[0128] (13)

[0129] Fusion loss:

[0130] The fusion loss consists of two parts: content loss and weight loss , can be represented as:

[0131] (14)

[0132] Content loss prompts the generator to produce a fused image with a similar data distribution to the source image, due to intensity loss. gradient loss and structural losses Composition can be represented as:

[0133] (15)

[0134] Intensity loss is used to measure the pixel-wise intensity difference between the fused result and the source image, and it is defined as:

[0135] (16)

[0136] in, and These are the height and width of the image, respectively. Represents the maximum operation per pixel. This represents the L1 norm.

[0137] Gradient loss is used to drive the fusion network to preserve texture detail information, and it is defined as follows:

[0138] (17)

[0139] in, This represents the Sobel gradient operator. This indicates the absolute value operation.

[0140] Considering the similarity of brightness, contrast, and structure across multiple scales between images, a Multi-Scale Structure Similarity Index (MS-SSIM) is used as the structural loss to preserve the brightness, contrast, and structural information in the source images. Structural Loss The definition of is:

[0141] (18)

[0142] in, This represents a function for calculating multi-scale structural similarity.

[0143] Weight loss Loss monitored by dual weights and complementary weight loss Composition, which is defined as:

[0144] (19)

[0145] The dual-weighted supervised loss is calculated based on the weight map generated from the WSM and is used to encourage the WSM to extract feature weights better and constrain FAF2M to generate better fused images. Its definition is:

[0146] (20)

[0147] Complementary loss, as an auxiliary loss, leverages the complementarity of different modal source images to encourage infrared and visible light WSM to focus on complementary regions between different modalities. It is defined as follows:

[0148] (twenty one)

[0149] Combat losses:

[0150] In DWSFusion, two discriminators are designed to force the fused image to retain more information from the source images. Therefore, the adversarial loss consists of three parts: generator loss. Infrared discriminator loss and visible light discriminator loss :

[0151] (twenty two)

[0152] The desired result is that the fused image generated by the generator scores as high as possible in both the infrared and visible light discriminators; therefore, the generator loss is:

[0153] (twenty three)

[0154] The goal is for the discriminator to correctly distinguish between the source image and the fused image. Therefore, it is desirable for the discriminator to score the source image as high as possible and the fused image as low as possible. Thus, the infrared discriminator loss and the visible light discriminator loss are designed as follows:

[0155] (twenty four)

[0156] (25)

[0157] in, It represents the mathematical expectation.

[0158] The dataset MSRS was used for training, with training images cropped to 240×320 patches. The fusion performance of DWSFusion on MSRS, TNO, RoadScene, and M3FD was fully validated. It was compared with nine other methods: IFCNN, SDNet, SeAFusion, DenseFuse, FusionGAN, GanMcC, TarDAL, CDDFuse, and Dif-Fusion. Six objective metrics were selected for quantitative evaluation: Mutual Information (MI), Standard Deviation (SD), Visual Information Fidelity (VIF), Structural Similarity Index Measure (SSIM), and edge-based similarity measure Q. AB / F And peak signal-to-noise ratio (PSNR).

[0159] The model in this embodiment is implemented in the PyTorch framework, with hyperparameters... and The initial learning rate was 0.1 and 10, respectively. The generator used the Adam optimizer, and the discriminator used the RMSprop optimizer. The initial learning rate, epoch, and batch size were set to 0.01, 100, and 64, respectively. All experiments were conducted on a platform equipped with an AMD EPYC 9754 CPU and an RTX 4090D (24GB) GPU.

[0160] The visualization results of different fusion algorithms on public datasets are as follows: Figures 3 to 6 As shown in the diagram. For each image, areas with complex texture details are magnified with red boxes, and salient targets are marked with green boxes. It can be seen that SDNet and GANMCC retain too much redundant information in the infrared images, resulting in poor visual effects of the fusion results. DenseFuse does not effectively highlight salient targets in infrared images. TarDAL highlights infrared targets most prominently, but also introduces too much redundant noise from the infrared images. In terms of texture details, only the method in this embodiment, SeAFusion, and CDDFuse do not exhibit artifacts in the red-boxed areas, achieving better visual effects. The visual results of the above four datasets demonstrate the superiority of the method of this invention in terms of fusion performance, effectively preserving sufficient texture details while highlighting salient targets, and suppressing the negative impact of redundant information on the fusion results. In addition, the method of this embodiment also fully preserves the color information of the visible light images.

[0161] Eight objective evaluation metrics were used to quantitatively analyze the above results, as shown in Table 1. It can be seen that the method in this embodiment exhibits superior performance across all metrics. On all datasets, DWSFusion's MI is significantly higher than other methods, and its VIF is the highest in TNO, RoadScene, and M3FD, while it lags slightly behind CDDFuse in MSRS. This indicates that the fused image generated by DWSFusion obtains sufficient and rich information from the source image and is highly consistent with the human visual system. DWSFusion's Q... AB / F Ranked first among the first three datasets and possessing excellent SD, DWSFusion demonstrates superior performance in preserving image edge information and exhibits high contrast. Furthermore, DWSFusion ranks third in SSIM on MSRS and fourth on the first two datasets, indicating excellent performance in preserving image structural information. While DWSFusion's PSNR is not high, this is attributed to the method's suppression of redundant information introduction. Subjective evaluation analysis shows that DWSFusion effectively suppresses noise information. Both qualitative and quantitative comparisons demonstrate the excellent fusion performance of the proposed method.

[0162] Table 1 Quantitative comparison of TNO datasets

[0163]

[0164] Efficiency Comparison: This embodiment compares the runtime and network parameter count of DWSFusion with nine other fusion methods. The results are shown in Table 2. It can be seen that DWSFusion has the best overall runtime efficiency and the fewest parameters. Compared to SeAFusion and CDDFuse, which have good fusion results, DWSFusion's parameter count is only 5.38% and 0.76% of these two methods, respectively. Compared to SDNet, which has the fewest parameters among the nine methods, it is only 13.43%. Based on the qualitative and quantitative analysis above, DWSFusion has higher fusion performance than other methods. In conclusion, DWSFusion achieves a better balance between runtime efficiency and fusion quality, achieving excellent fusion performance with very few parameters.

[0165] Table 2 Comparison of running time and parameter count for different methods

[0166]

[0167] In this embodiment, the effectiveness of different modules is verified through qualitative and quantitative analysis. In the qualitative analysis, prominent targets are marked with green boxes, and areas with complex texture details are marked and magnified with red boxes, such as... As shown in Table 3, quantitative analysis results are presented with and without the use of the Frequency Adaptive Attention Module (FA2M), Cross-Weight Strategy (CW), Dual-Weight Supervision (DWS), and Dual Discriminator (DD).

[0168] DWSFusion on MI, SD, VIF, and Q in two datasets AB / F All metrics achieved optimal values. The absence of FA2M leads to reduced preservation of image edge information and artifacts in the fusion result. The absence of CW results in a significant decrease in the information transferred from the source image to the fusion result, especially information about salient infrared targets. Combined with... Analysis shows that FA2M and CW can retain useful information while suppressing redundant information. DWS can improve the contrast of the fusion result, prompting the network to generate a better fused image. DD's impact on the network is reflected in various aspects; without DD's constraint, the overall fusion performance will decline.

[0169] Table 3 Quantitative Comparison of Ablation Experiments

[0170]

[0171] To further verify the performance of DWSFusion in advanced vision tasks, the fusion results of all the above methods were applied to object detection and semantic segmentation tasks, respectively. The test set consisted of 80 pairs of images of people and vehicles labeled in the MSRS dataset. YOLOv5s was used as the detector and mean average precision (mAP) was used as the evaluation metric. DeepLabV3Plus was used as the semantic segmentation model and mean intersection over union (mIoU) was used as the evaluation metric. The experimental results are shown in Table 4.

[0172] Table 4 Quantitative Comparison of Applications in Advanced Vision Tasks

[0173]

[0174] Visualization results of the object detection task are as follows As shown, this example depicts a nighttime scene containing pedestrians and vehicles. In the infrared image, vehicles were not detected due to partial occlusion, while in the visible light image, neither pedestrians nor vehicles were detected due to poor lighting conditions. DWSFusion effectively fuses information from different modalities, allowing the detector to detect all objects from the fused image with higher confidence compared to other methods. Quantitative analysis also shows that compared to SeAFusion and CDDFuse, which have better fusion results, DWSFusion maintains good performance in detection tasks with very few parameters.

[0175] Visualization results of semantic segmentation tasks are as follows As shown in the figure, SDNet and GANMcC perform poorly in segmentation, for example, misclassifying large areas such as "roads" and "terrain," while GANMcC and TarDAL miss segmentation results for smaller targets such as "bicycles" and "traffic signs." In terms of visual performance, SeAFusion, DenseFuse, CDDFuse, and DWSFusion achieve relatively good segmentation results. Combined with the analysis in Table 4, the method ranked third in overall segmentation performance, second only to SeAFusion. In conclusion, it still performs excellently in advanced vision tasks, achieving performance metrics and visual effects similar to the best methods with only a very small number of parameters.

[0176] This embodiment proposes a lightweight IVIF network, DWSFusion, based on dual-weight supervision. The principle is as follows: First, the weight maps of the infrared and visible light images are extracted in the generator, respectively. A cross-weighting strategy guides the feature fusion module to generate a fused image. Second, the dual-weight supervision constraint strategy utilizes the generator's WSM to extract infrared and visible light weight maps from the fused image, respectively, and calculates the loss by comparing these weight maps with the corresponding weight maps of the source images. This improves the WSM's feature extraction capability while prompting the feature fusion module to retain useful information from the source images and ignore redundant information. Finally, a dual discriminator with a WGAN-GP structure is used to constrain the fused image to maintain the same information distribution as the source images. This invention achieves optimal fusion performance with a minimal number of parameters and also demonstrates excellent performance in advanced computer vision tasks.

[0177] Specifically, the following steps are included:

[0178] S1. Pre-train the WSM module and use this module to extract features from infrared and visible light images respectively;

[0179] S2. Use a cross-weighting strategy to guide the feature fusion module in generating a fused image;

[0180] S3. Using the WSM module of S1, extract the fused infrared features and visible light image features from the fused image respectively;

[0181] S4. Use a dual discriminator with a WGAN-GP structure to constrain the fused image to maintain the same information distribution as the source image;

[0182] S5. Use fusion loss and adversarial loss to constrain model convergence, forcing the generator to produce better fused images.

[0183] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0184] In this disclosure, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The block diagrams of devices, apparatuses, devices, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0185] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0186] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0187] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0188] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0189] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A dual-weighted supervised infrared and visible light image fusion system, characterized in that, include: A generator based on a weighted supervision module, a constraint strategy based on dual weighted supervision, and a discriminator; The generator based on the weighted supervision module is used to generate a fused image from infrared and visible light images; The dual-weight supervision constraint strategy is used to improve the feature extraction and fusion performance of the generator; The discriminator is used to ensure that the fused image and the source image maintain the same information distribution.

2. The infrared and visible light image fusion system based on dual-weight supervision according to claim 1, characterized in that, The generator based on the weight supervision module includes: a weight supervision module based on frequency adaptive attention, a cross-weight strategy, and a multi-scale feature fusion module; The frequency-adaptive attention-based weight supervision module is used to convert infrared images and visible light images into weight maps; The cross-weight strategy is used to calculate the feature map from the weight map; The multi-scale feature fusion module is used to fuse feature maps to obtain a fused image.

3. The infrared and visible light image fusion system based on dual-weight supervision according to claim 2, characterized in that, The method for converting infrared and visible light images into weighted maps includes: The source image is downsampled twice, and then shallow features are obtained based on the results of the two downsampling. The shallow features are then downsampled twice more to obtain deep features. The deep features are upsampled twice using transposed convolutions, and the results of the two upsamplings are concatenated with the dimensions of the shallow features to obtain a weighted feature that contains both shallow and deep semantic features. Then, the weighted feature is upsampled twice to obtain the weighted feature. A weight map is obtained by adding the source image and the weight features pixel by pixel using skip connections.

4. The infrared and visible light image fusion system based on dual-weight supervision according to claim 3, characterized in that, The method for calculating the feature map from the weight map includes: Feature maps of infrared and visible light images are obtained by calculating based on the weight map and adaptive weight balancing parameters.

5. The infrared and visible light image fusion system based on dual-weight supervision according to claim 4, characterized in that, The adaptive weight balancing parameters are calculated by a module consisting of convolutional, pooling, and fully connected layers.

6. The infrared and visible light image fusion system based on dual-weight supervision according to claim 5, characterized in that, The method for fusing feature maps to obtain a fused image includes: The infrared and visible light features are spliced ​​together and then expanded into channels to adjust the weights of features in different frequency ranges.

7. The infrared and visible light image fusion system based on dual-weight supervision according to claim 6, characterized in that, The dual-weight supervision constraint strategy is used to constrain the frequency-adaptive attention-based weight supervision module to extract information from the source image, and to encourage the multi-scale feature fusion module to retain useful information while suppressing the influence of redundant information on the fusion result.

8. The infrared and visible light image fusion system based on dual-weight supervision according to claim 7, characterized in that, The discriminator adopts a WGAN-GP structure. The loss function of the WGAN-GP structure is constructed based on the distribution of real data, the distribution of generated data, the distribution of interpolation between real and generated data, and the weight coefficients of the gradient penalty term.

9. The infrared and visible light image fusion system based on dual-weight supervision according to claim 8, characterized in that, The loss function of the discriminator includes fusion loss and adversarial loss; The fusion loss includes content loss and weight loss; The content loss includes intensity loss, gradient loss, and structure loss; The weight loss includes dual-weight monitoring loss and complementary weight loss; The adversarial loss includes generator loss, infrared discriminator loss, and visible light discriminator loss.

10. The infrared and visible light image fusion system based on dual-weight supervision according to claim 9, characterized in that, The content loss is used to prompt the generator to generate a fused image with a similar data distribution to the source image; The intensity loss is used to measure the pixel-by-pixel intensity difference between the fusion result and the source image, and the gradient loss is used to drive the fusion network to have the ability to preserve texture detail information; A multi-scale structural similarity index is used as the structural loss to preserve the brightness, contrast and structural information in the source image; The dual-weight supervision loss is calculated from the weight map generated by the frequency-adaptive attention-based weight supervision module. It is used to encourage the frequency-adaptive attention-based weight supervision module to extract feature weights and to constrain the multi-scale feature fusion module to generate a fused image. The complementary loss is used to leverage the complementarity of source images from different modalities to encourage a frequency-adaptive attention-based weighted supervision module to focus on complementary regions between different modalities.

Images