Infrared and visible image fusion method based on multi-discriminator generative adversarial network

Abstract

The application discloses an infrared and visible light image fusion method based on a multi-discriminator generative adversarial network, and the main process can be divided into the following steps: calculating and preprocessing a difference image; taking a source image (an infrared image and Iir a visible light image Ivis ) and the difference image (an infrared difference image Idif‑ir and a visible light difference image Idif‑vis ) as input to train a network model (a generator model and a discriminator model); and generating a fusion image by using the trained generator model. Thus, in the actual application of infrared and visible light image fusion, not only the thermal radiation information in the infrared image can be fully retained, but also the texture details in the visible light image can be effectively reproduced.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a method for fusing infrared and visible light images based on a multi-discriminator generative adversarial network. Background Technology

[0002] Image fusion aims to combine different information from images obtained by multiple sensors into a single image to meet various application requirements. Infrared images contain rich thermal radiation information and can be distinguished based on the different thermal feedback between the target and the background, but they are usually low in resolution and lack texture details. Visible light images have high resolution and rich details, closely matching human visual perception, but are easily affected by external factors. The fused image obtained using image fusion technology combines the advantages of both types of images.

[0003] In existing multimodal image fusion research, the fusion of infrared and visible light images is a key branch. Researchers have proposed different fusion methods and strategies to address its characteristics. Based on the machine learning techniques used in the fusion strategies, fusion methods can be broadly categorized into fusion methods based on traditional algorithms and fusion methods based on deep learning. Common fusion methods based on traditional algorithms include: fusion methods based on multi-scale transformations, fusion methods based on sparse representations, and fusion methods based on subspaces. Fusion methods based on traditional algorithms often apply the same transformation to different images, failing to extract key features specifically tailored to the characteristics of each image. Furthermore, these methods cannot effectively preserve a wide variety of detailed information, thus affecting the fusion results. In contrast, fusion methods based on deep learning can effectively solve these problems. Common deep learning-based fusion methods include: fusion methods based on autoencoders (AEs), fusion methods based on convolutional neural networks (CNNs), and fusion methods based on generative adversarial networks (GANs). In existing fusion methods, image fusion based on infrared (AE) requires training the encoder and decoder in the network on public datasets to achieve optimal performance. These methods rely on pre-set feature fusion strategies, which to some extent limits the effectiveness of infrared and visible light image fusion. In CNN-based infrared and visible light image fusion methods, ground truth values ​​for training the deep learning model are needed beforehand. However, ground truth values ​​do not exist in infrared and visible light image fusion tasks, and the evaluation of the fused image often relies on subjective human visual assessment supplemented by objective evaluation metrics. Therefore, CNN-based fusion strategies are limited, affecting the fusion results. GAN-based fusion strategies use an adversarial game between the generator and discriminator for unsupervised training. The generator aims to generate images with infrared intensity and added visible gradients, while the discriminator aims to distinguish the generated image from the source image, ensuring the final fused image possesses both the clear thermal radiation intensity of the infrared image and the texture details of the visible light image. GAN-based fusion strategies not only overcome the shortcomings of the above methods but are also better suited for the task of fusing infrared and visible light images.

[0004] In existing GAN-based infrared and visible light image fusion methods, some researchers use a single generator-single discriminator structure, where the discriminator distinguishes between the fused image and the visible light image to guide the generator to retain as much texture detail as possible from the visible light image. To address the information imbalance problem in the fusion result caused by a single discriminator, some researchers have proposed a single generator-dual discriminator structure, using dual discriminators to distinguish between the source images of the two modalities. Furthermore, some researchers have introduced a differential discriminator based on existing research, proposing a single generator-triple discriminator structure, using the differential image as an additional input to the network, thus improving fusion performance. For the proposed GAN models used for image fusion, increasing the number of discriminators can constrain the generator from multiple angles, improving fusion performance to some extent. In addition, the differential image can focus on the unique information of the source image, helping the image fusion network model retain more source image information. However, most existing methods perform a "thresholding" operation (thresholding to 0) or take the absolute value of the differential image to avoid negative grayscale values. In practice, thresholding will lose some information from the source image; while taking the absolute value will preserve all the information, it will not highlight the unique information of the multimodal image.

[0005] To address the problems of existing GAN-based methods in fusing infrared and visible light images, this invention proposes a novel infrared and visible light image fusion method based on a multi-discriminator generative adversarial network (GAN). The proposed network model employs a single generator-four discriminators structure, adding two differential discriminators to the frontier algorithm to establish adversarial training with the generator, thereby further constraining the generator's optimization tendency. First, the generator uses a dual encoder-single decoder structure, where different encoders extract features from different modalities, and the decoder reconstructs the fused image based on the fusion features. Second, unlike other methods that perform absolute value operations on the differential images, this invention normalizes the differential images to highlight the unique information of each modality. Finally, to avoid the problem of generator convergence difficulties due to excessive discriminator constraints, the loss function adopts a strategy of primarily using source image loss and secondarily using differential image loss. Experimental results on public datasets show that the proposed algorithm not only fully preserves the thermal radiation information in infrared images but also effectively reproduces the texture details in visible light images. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing an infrared and visible light image fusion method based on a multi-discriminator generative adversarial network, thereby solving the problems mentioned in the background.

[0007] To achieve the above objectives, the present invention provides the following technical solution: an infrared and visible light image fusion method based on a multi-discriminator generative adversarial network, characterized in that: the infrared and visible light image fusion is completed according to steps 1 to 3 as follows:

[0008] Step 1: Calculation and preprocessing of the differential image, converting the infrared image I... ir With visible light image I vis The differences are calculated and normalized separately to obtain the difference image I. dif-ir with I dif-vis Step 2: Train the network model using the source image and the difference image as input. The training process consists of steps 2-1 to 2-4.

[0009] Step 2-1: Transfer the infrared image I ir With difference image I dif-ir Link, visible light image I vis With difference image I dif-vis The connection serves as the input to the different encoders in the generator G in step 2-2;

[0010] Step 2-2: Generator G extracts and fuses features from the data in Step 2-1, and then reconstructs the fused image I based on the fused features. F ;

[0011] Steps 2-3: Combine the fused image with the infrared image I respectively. ir Visible light image I vis and difference image I dif-ir ,I dif-vis Input to discriminator (D) ir D vis D dif-ir With D dif-vis In this process, adversarial training is established with the generator G;

[0012] Step 2-4: Iterate through steps 2-1 to 2-3. When the adversarial training reaches equilibrium, i.e. the discriminator can no longer distinguish whether the input sample comes from an image generated by the generator or a real image, terminate the training to obtain the generator G required for fusion.

[0013] Step 3: Generate the fused image using the trained generator model G. Specifically, the infrared and visible light images are concatenated with the two difference images respectively, and then input into the generator trained in Step 2 to obtain the final fused result.

[0014] As a preferred embodiment of the present invention, the specific steps for calculating the difference image in step 1 are as follows:

[0015] Step 1-1: Using infrared image Iir Subtract visible light image I vis This yields an infrared differential image I that emphasizes the intensity of thermal radiation. dif-ir ;

[0016] Steps 1-2: Using visible light image I vis Subtract infrared image I ir This yields a visible light difference image I that highlights texture details. dif-vis .

[0017] As a preferred embodiment of the present invention: the normalization process in the preprocessing of the difference image in step 1 controls the pixel grayscale value between 0 and 1, specifically as follows:

[0018]

[0019] Where v(i,j) is the gray value of the pixel at (i,j) in the difference image, v min With v max These represent the minimum and maximum gray values ​​in the difference image, respectively.

[0020] As a preferred technical solution of the present invention: the generator network G in step 2-2 adopts a dual encoder-single decoder structure, specifically: first, the source image and the differential image highlighting the unique information of the modality image are used as inputs to the branch encoders; second, the two branch encoders are respectively responsible for extracting the features of the infrared image and the visible light image; finally, the high-dimensional features of the different modal images are concatenated and input into the decoder to reconstruct the fused image.

[0021] As a preferred technical solution of the present invention: the four discriminators in steps 2-3 adopt the same network structure, which includes five layers of convolutional neural network in sequence. In the first four layers, a convolutional kernel with a size of 3×3 is used and the stride is set to 2. In the second to fourth layers, a batch normalization layer is added. In the last layer, a fully connected layer is first used to integrate the features extracted by the convolutional layer, and then the Tanh activation function is used to calculate the scalar.

[0022] As a preferred embodiment of the present invention: during the iterative training in steps 2-4, a loss function is required to evaluate the model's prediction discrepancies. This loss function consists of a generator loss function and a discriminator loss function, wherein the generator loss function L... G Mainly composed of countermeasures loss L adv With content loss L content It consists of two parts, used to feed back the generator network training loss; the four discriminators use similar loss functions L. D The discriminator's judgment on the input result is fed back to the generator, establishing adversarial training with the generator. The specific calculation formula is as follows:

[0023] L G =L adv +λL content

[0024]

[0025] Where λ is the weight parameter. and Each corresponds to one of the four discriminators D. ir D vis D dif-ir With D dif-vis ;

[0026] Specifically, the adversarial loss L adv Primarily used to constrain the optimization direction of the generator, its formula is defined as:

[0027] L adv =E[log(1-D ir (I F ))]+E[log(1-D vis (I F ))]+E[log(1-D dif-ir (I F ))]+E[log(1-D dif-vis (I F ))]

[0028] Where E[·] is the expectation, and D(·) is the probability of the discriminator classifying the input image;

[0029] Specifically, the content loss L content By comparing the differences between the fused image and the input image, the generator is guided to produce a fused result that simultaneously preserves the thermal radiation information of the infrared image and the texture information of the visible light image. The formula is defined as follows:

[0030] L content =αL int +βL grad +γL SSIM

[0031] Where α, β, and γ are weight parameters, L int For strength loss, L grad For gradient loss, L SSIM For structural similarity loss; L int L grad , and L SSIM They are defined as follows:

[0032]

[0033]

[0034] L SSIM =ω·L SSIM-img +(1-ω)·L SSIM-dif

[0035] Where ω is the weight parameter, H,W are the dimensions of the input image, and L is the weight parameter. int-img For the source image intensity loss, L int-dif For differential image intensity loss, L grad-img For the source image gradient loss, L grad-dif For the differential image gradient loss, L SSIM-img For source image structural similarity loss, L SSIM-dif For differential image structural similarity loss, L SSIM (·) represents the similarity between two images; L int-img L int-dif L grad-img L grad-dif L SSIM-img , and L SSIM-dif The definitions are as follows:

[0036] L int-img =a·||I F -I ir || F +(1-a)·||I F -I vis || F

[0037] L int-dif =a·||I F -I dif-ir || F +(1-a)·||I F -I dif-vis || F

[0038]

[0039]

[0040] L SSIM-img =(1-L) SSIM (I F ,I ir ))+(1-L SSIM (I F ,I vis ))

[0041] L SSIM-dif =(1-L) SSIM (I F ,I dif-ir ))+(1-L SSIM (IF ,I dif-vis ))

[0042] Specifically, the loss function of each discriminator is defined as follows:

[0043]

[0044]

[0045]

[0046]

[0047] Among them, discriminator D ir With D vis The input image is the source image (I ir with I vis ) or fused image (I F Discriminator D dif-ir With D dif-vis The input image is a difference image (I dif-ir with I dif-vis ) or fused image (I F ).

[0048] As a preferred embodiment of the present invention: In the dual encoder-single decoder structure, the dual encoder structure includes two branches, which are used to extract infrared thermal radiation intensity and visible light texture features respectively. Each branch consists of four convolutional layers, densely connected using a DenseNet structure. Specifically, the first layer consists of a 3×3 convolutional kernel, a switchable normalization layer, and a Leaky ReLU activation function; the last three layers add a Convolutional Block Attention (CBAM) module, and the number of channels in all convolutional layers is set to 64, and the stride of the convolutional kernel is set to 1; in addition, CBAM is introduced to improve feature extraction capability, which mainly consists of steps A to F:

[0049] Step A: The feature map is first input into the channel attention module. Based on the width and height of the input feature map, global max pooling and global average pooling are performed to obtain two feature maps.

[0050] Step B: Input the two feature maps into the parameter-shared multilayer perceptron to generate their respective channel attention feature maps, and then perform element-wise summation and Sigmoid activation function operation to obtain the final channel attention feature map;

[0051] Step C: Multiply the original input features element-wise with the channel attention feature map, and input the result into the spatial attention module;

[0052] Step D: Perform max pooling and global average pooling on the channel dimension of the feature map input to the spatial attention module to obtain two feature maps;

[0053] Step E: Connect them based on the channel dimension and perform convolution through a convolutional layer, then generate a spatial attention map by operating the sigmoid activation function;

[0054] Step F: Multiply the input vector of the spatial attention module element-wise with the spatial attention map to obtain the final output features.

[0055] As a preferred technical solution of the present invention: in the dual encoder-single decoder structure, the single decoder network structure consists of two convolutional layers, wherein the first convolutional layer consists of a 3×3 convolutional kernel, a switchable normalization layer and the activation function Leaky ReLU, and the second layer consists of a 3×3 convolutional kernel and the Tanh activation function.

[0056] The infrared and visible light image fusion method based on multi-discriminator generative adversarial networks described in this invention has the following technical advantages compared with existing technologies:

[0057] The beneficial effects of this invention are as follows: The generative adversarial network fusion framework proposed in this invention includes one generator and four discriminators, using difference images as auxiliary information to further improve the fusion performance of the network. In the proposed method, the difference images not only serve as additional information of the source images, guiding the generator to focus on the unique information of different modal images, but also serve as the real data distribution, assisting the difference discriminators and generator in adversarial training. In the proposed network model, the generator adopts a dual encoder-single decoder structure, where the encoder aims to extract features of different modalities, mainly using a dense connection structure combined with an attention module; the decoder aims to reconstruct the fused image based on the high-dimensional features of the connections. The discriminators determine whether the input image comes from a real image or an image generated by the generator, and perform constraint optimization on the generator based on the determination result. Attached Figure Description

[0058] Figure 1 This is a flowchart of the infrared and visible light image fusion method based on multi-discriminator generative adversarial networks described in this invention.

[0059] Figure 2 This is the overall fusion framework of the infrared and visible light image fusion method based on multi-discriminator generative adversarial networks described in this invention;

[0060] Figure 3 A differential image and its preprocessing effect diagram provided by the method of the present invention;

[0061] Figure 4A diagram illustrating a generator network structure for the method provided by this invention;

[0062] Figure 5 A schematic diagram illustrating a Convolutional Block Attention Module (CBAM) structure for the method provided by this invention;

[0063] Figure 6 This is a case study diagram of a discriminator network structure for the method provided by the present invention; Detailed Implementation

[0064] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0065] Example: Please refer to Figure 1 This invention provides a technical solution: an infrared and visible light image fusion method based on a multi-discriminator generative adversarial network, which completes the infrared and visible light image fusion according to steps 1 to 3: Step 1: calculation and preprocessing of the difference image; Step 2: training the network model with the source image and the difference image as input; Step 3: generating the fused image using the trained generator model.

[0066] Experimental group: Please refer to Figure 2 The specific implementation steps of the infrared and visible light image fusion method based on multi-discriminator generative adversarial networks are as follows:

[0067] like Figure 3 As shown, perform step 1 using infrared image I. ir Subtract visible light image I vis The resulting infrared differential image can emphasize the intensity of thermal radiation, such as Figure 3 (c); Using visible light image I vis Subtract infrared image I ir The resulting visible light difference image can highlight texture details, such as Figure 3 (d); where the difference image is normalized to keep the pixel grayscale value between 0 and 1, as shown in the following formula:

[0068]

[0069] Where v(i,j) is the gray value of the pixel at (i,j) in the difference image, v min With v max These represent the minimum and maximum gray values ​​in the difference image, respectively. Figure 3 (e) and (f) are the normalized infrared difference images I. dif-ir Differential image with visible light I dif-vis .

[0070] Perform the training task in step 2 according to steps 2-1 to 2-4:

[0071] Step 2-1: Transfer the infrared image I ir With difference image I dif-ir Link, connect the visible light image I vis With difference image I dif-vis The connection serves as the input to the different encoders in the generator G in step 2-2;

[0072] Step 2-2: Generator G extracts and fuses features from the data in Step 2-1, and then reconstructs the fused image I based on the fused features. F As a specific embodiment of the present invention, the specific network structure of its generator network is as follows: Figure 4 As shown, the dual encoder structure comprises two branches, one for extracting infrared thermal radiation intensity and the other for extracting visible light texture features. Each branch consists of four convolutional layers, densely connected using a DenseNet structure to fully utilize multi-layer features. Specifically, the first layer extracts shallow features of the image, consisting of a 3×3 convolutional kernel, a switchable normalization layer, and the Leaky ReLU activation function. The latter three layers extract depth features of the image, and their network structure adds a convolutional block attention module to the first layer. All convolutional layers have 64 channels, and the kernel stride is set to 1.

[0073] In the deep feature layer of the encoder, CBAM is introduced to enhance feature extraction capabilities. CBAM consists of two sub-modules: a channel attention module and a spatial attention module. The channel attention module describes the relationship between each channel, while the spatial attention module describes the spatial relationship between the deep features. Its structure is as follows: Figure 5 As shown. Specifically, 1) The feature map is first input into the channel attention module. Based on the width and height of the input feature map, global max pooling and global average pooling are performed to obtain two feature maps. The two feature maps are then input into a multilayer perceptron with shared parameters to generate their respective channel attention feature maps. Then, element-wise summation and Sigmoid activation function operation are performed to obtain the final channel attention feature map. 2) The original input feature map and the channel attention feature map are multiplied element-wise, and the result is input into the spatial attention module. Max pooling and global average pooling are performed on the channel dimension of the feature map input into the spatial attention module to obtain two feature maps. Then, they are concatenated based on the channel dimension and convolved through a convolutional layer, and then the Sigmoid activation function operation is performed to generate the spatial attention map. Finally, the input vector of the spatial attention module and the spatial attention map are multiplied element-wise to obtain the final output feature.

[0074] After the two-branch encoders extract features from source images of different modalities, these features are concatenated based on the channel dimension and input into the decoder. The decoder reconstructs the fused image based on the concatenated high-dimensional features, and its network structure consists of two convolutional layers. The first convolutional layer consists of a 3×3 convolutional kernel, a switchable normalization layer, and the LeakyReLU activation function, while the second layer consists of a 3×3 convolutional kernel and the Tanh activation function.

[0075] Steps 2-3: Combine the fused image with the infrared image I respectively. ir Visible light image I vis and difference image I dif-ir ,I dif-vis Input to discriminator (D) ir D vis D dif-ir With D dif-vis In this process, adversarial training is established with the generator G;

[0076] As a specific embodiment of the present invention, the four discriminators adopt the same network structure, mainly composed of a five-layer convolutional neural network, as shown in the specific network structure. Figure 6 As shown. Specifically, a 3×3 convolutional kernel with a stride of 2 is used in the first four layers; batch normalization layers are added in the second to fourth layers; and in the last layer, a fully connected layer (FC) is used to integrate the features extracted by the convolutional layers. Then, a scalar is calculated using the Tanh activation function to reflect the probability that the discriminator judges the input image to be from the source image or the difference image rather than the fused image.

[0077] Step 2-4: Iterate through steps 2-1 to 2-3. When the adversarial training reaches equilibrium, i.e. the discriminator can no longer distinguish whether the input sample comes from an image generated by the generator or from a real image, terminate the training.

[0078] The loss function consists of the generator loss function and the discriminator loss function, where the generator loss function L... G Mainly composed of countermeasures loss L adv With content loss L content It consists of two parts, used to feed back the generator network training loss; the four discriminators use similar loss functions L. D The discriminator's judgment on the input result is fed back to the generator, establishing adversarial training with the generator. The specific calculation formula is as follows:

[0079] L G =L adv +λL content

[0080]

[0081] Where λ is the weight parameter. and Each corresponds to one of the four discriminators D. ir D vis D dif-ir With D dif-vis .

[0082] Combat loss L adv Primarily used to constrain the optimization direction of the generator, its formula is defined as:

[0083] L adv =E[log(1-D ir (I F ))]+E[log(1-D vis (I F ))]+E[log(1-D dif-ir (I F ))]+E[log(1-D dif-vis (I F ))]

[0084] Where E[·] is the expectation and D(·) is the probability that the discriminator classifies the input image.

[0085] Content loss L content By comparing the differences between the fused image and the input image, the generator is guided to produce a fused result that simultaneously preserves the thermal radiation information of the infrared image and the texture information of the visible light image. The formula is defined as follows:

[0086] L content =αL int +βL grad +γL SSIM

[0087] Where α, β, and γ are weight parameters, L int For strength loss, L grad For gradient loss, L SSIM For structural similarity loss; L int L grad , and L SSIM They are defined as follows:

[0088]

[0089]

[0090] L SSIM =ω·L SSIM-img +(1-ω)·L SSIM-dif

[0091] Where ω is the weight parameter, H,W are the dimensions of the input image, and L is the weight parameter.int-img For the source image intensity loss, L int-dif For differential image intensity loss, L grad-img For the source image gradient loss, L grad-dif For the differential image gradient loss, L SSIM-img For source image structural similarity loss, L SSIM-dif For differential image structural similarity loss, L SSIM (·) represents the similarity between two images; L int-img L int-dif L grad-img L grad-dif L SSIM-img , and L SSIM-dif The definitions are as follows:

[0092] L int-img =a·||I F -I ir || F +(1-a)·||I F -I vis || F

[0093] L int-dif =a·||I F -I dif-ir || F +(1-a)·||I F -I dif-vis || F

[0094]

[0095]

[0096] L SSIM-img =(1-L) SSIM (I F ,I ir ))+(1-L SSIM (I F ,I vis ))

[0097] L SSIM-dif =(1-L) SSIM (I F ,I dif-ir ))+(1-L SSIM (I F ,I dif-vis ))

[0098] The loss function of each discriminator is defined as follows:

[0099]

[0100]

[0101]

[0102]

[0103] Among them, discriminator D ir With D vis The input image is the source image (I ir with I vis ) or fused image (I F Discriminator D dif-ir With D dif-vis The input image is a difference image (I dif-ir with I dif-vis ) or fused image (I F ).

[0104] Step 3: Generate the fused image using the trained generator model. Specifically, the infrared and visible light images are concatenated with the two difference images respectively, and then input into the generator trained in Step 2 to obtain the final fused result.

[0105] Experimental Conclusions: This invention proposes an infrared and visible light image fusion method based on a multi-discriminator generative adversarial network (GAN), which is an "end-to-end" network model consisting of one generator and four discriminators. Experiments were conducted using publicly available infrared and visible light image datasets. The results show that, compared to existing methods, the fusion result obtained by the algorithm of this invention has richer texture information and better subjective visual effects. Furthermore, objective evaluation results show that the algorithm of this invention outperforms the average values ​​of the comparative methods in terms of information entropy, average gradient, correlation coefficient, and difference correlation index by approximately 6.02%, 25.93%, 7.61%, and 16.77%, respectively. This demonstrates that the proposed method effectively preserves the thermal radiation information of infrared images while better fusing the texture information of visible light images, thereby improving the performance of existing infrared and visible light image fusion algorithms.

[0106] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for fusing infrared and visible light images based on multi-discriminator generative adversarial networks, characterized in that: This includes completing the fusion of infrared and visible light images according to steps 1 to 3 below: Step 1: Calculation and preprocessing of differential images, specifically the infrared image. With visible light images The differences are calculated and normalized separately to obtain the difference images. and ; Step 2: Train the network model using the source image and the difference image as input. The training process consists of steps 2-1 to 2-4: Step 2-1: Transfer the infrared image With difference image Link, visible light image With difference image The link serves as the generator in step 2-2. The inputs of different encoders; Step 2-2: Generator The data from step 2-1 are subjected to feature extraction and fusion, and then the fused image is reconstructed based on the fused features. ; Steps 2-3: Merge the images Compared with infrared images Visible light images and difference image , Input to discriminator , , and In, with generator Establish adversarial training; Step 2-4: Iterate through steps 2-1 to 2-3. When the adversarial training reaches equilibrium, i.e., the discriminator can no longer distinguish whether the input sample comes from an image generated by the generator or a real image, terminate the training to obtain the generator required for fusion. ; During iterative training, a loss function is needed to evaluate the model's predictive discrepancies. This loss function consists of a generator loss function and a discriminator loss function, where the generator loss function... Mainly due to the loss of the opponent With content loss It consists of two parts, used to feed back the generator network training loss; the four discriminators use similar loss functions. The discriminator's judgment on the input result is fed back to the generator, establishing adversarial training with the generator. The specific calculation formula is as follows: ； ； in For weight parameters, , , and Each corresponds to one of the four discriminators. , , and ; Specifically, the adversarial loss Primarily used to constrain the optimization direction of the generator, its formula is defined as: ； in As expected, The probability of the discriminator classifying the input image; Specifically, the content loss By comparing the differences between the fused image and the input image, the generator is guided to produce a fused result that simultaneously preserves the thermal radiation information of the infrared image and the texture information of the visible light image. The formula is defined as follows: ； in , , For weight parameters, For strength loss, For gradient loss, For structural similarity loss; Step 3: Use the trained generator To generate a fused image, specifically, the infrared and visible light images are concatenated with the two difference images respectively, and then input into the generator trained in step 2 to obtain the final fusion result.

2. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 1, characterized in that: The specific steps for calculating the difference image in step 1 are as follows: Step 1-1: Using Infrared Images Subtract visible light image This yields an infrared differential image that emphasizes the intensity of thermal radiation. ; Steps 1-2: Using visible light images Subtract infrared image This yields a visible light difference image that highlights texture details. .

3. The infrared and visible light image fusion method based on multi-discriminator generative adversarial networks according to claim 1, characterized in that: The normalization process in the preprocessing of the difference image in step 1 controls the pixel grayscale value between 0 and 1, as shown in the following formula: ； in For difference images in The grayscale value of the pixel. and These represent the minimum and maximum gray values ​​in the difference image, respectively.

4. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 1, characterized in that: generator network in step 2-2 It adopts a dual encoder-single decoder structure.

5. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 1, characterized in that: In steps 2-3, the four discriminators use the same network structure, consisting of five layers of convolutional neural networks, with the first four layers using a size of [missing information]. The convolutional kernel is set with a stride of 2; batch normalization layers are added in the second to fourth layers; in the last layer, fully connected layers are first used to integrate the features extracted by the convolutional layers, and then the scalar is calculated using the Tanh activation function.

6. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 1, characterized in that: , ,and They are defined as follows: ； ； ； in Here, H and W are the weight parameters, and H and W are the dimensions of the input image. For the source image intensity loss, For differential image intensity loss, For source image gradient loss, For differential image gradient loss, For source image structural similarity loss, For difference image structural similarity loss, The similarity between two images; , , , , ,and The definitions are as follows: ； ； ； ； ； ； Specifically, the loss function of each discriminator is defined as follows: ； ； ； ； Among them, the discriminator and The input image is the source image. and or merged images Discriminator and The input image is a difference image. and or merged images .

7. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 4, characterized in that: In the described dual encoder-single decoder structure, the dual encoder structure includes two branches, used to extract infrared thermal radiation intensity and visible light texture features respectively. Each branch consists of four convolutional layers, densely connected using a DenseNet structure. Specifically, the first layer of the network consists of a size of... The convolutional kernels, switchable normalization layers, and Leaky ReLU activation function constitute the core. The last three layers add a convolutional block attention module. All convolutional layers have 64 channels, and the kernel stride is set to 1. It should be noted that the convolutional block attention module is introduced to improve feature extraction capabilities, and it mainly consists of steps A to F. Step A: The feature map is first input into the channel attention module. Based on the width and height of the input feature map, global max pooling and global average pooling are performed to obtain two feature maps. Step B: Input the two feature maps into the parameter-shared multilayer perceptron to generate their respective channel attention feature maps, and then perform element-wise summation and Sigmoid activation function operation to obtain the final channel attention feature map; Step C: Multiply the original input features element-wise with the channel attention feature map, and input the result into the spatial attention module; Step D: Perform max pooling and global average pooling on the channel dimension of the feature map input to the spatial attention module to obtain two feature maps; Step E: Connect them based on the channel dimension and perform convolution through a convolutional layer, then generate a spatial attention map by operating the sigmoid activation function; Step F: Multiply the input vector of the spatial attention module element-wise with the spatial attention map to obtain the final output features.

8. The infrared and visible light image fusion method based on a multi-discriminator generative adversarial network according to claim 4, characterized in that: In the dual encoder-single decoder structure, the single decoder network structure consists of two convolutional layers, wherein the first convolutional layer has a size of The convolutional kernel consists of a switchable normalization layer and the Leaky ReLU activation function. The second layer is composed of... It consists of a convolution kernel and a Tanh activation function.

Images