Infrared and visible light image fusion method and system based on pseudo-supervised generative adversarial network, and medium

Abstract

The embodiment of the invention provides an infrared and visible light image fusion method and system based on a pseudo-supervised generative adversarial network, and a medium, and the method comprises the steps: inputting an infrared image and a visible light image, carrying out the foreground and background segmentation, selecting a distance measurement mode based on the infrared segmentation image and the visible light segmentation image, and calculating the distance, performing weighting processing on the infrared distance image and the visible light distance image, and fusing the processed images to obtain a real pseudo ground image; performing analysis and fusion enhancement on the infrared image and the visible light image based on a pseudo-supervised generative adversarial network to obtain an enhanced fusion image; performing comparative analysis on the enhanced fusion image and the pseudo ground real image to obtain a judgment result, and adjusting parameters of the pseudo-supervised generative adversarial network based on the judgment result; by introducing a pseudo ground real image and a pseudo supervision generative adversarial network, the image is subjected to fusion weight fusion, and network parameters are dynamically adjusted, so that the quality of the fused image is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an infrared and visible light image fusion method and system based on pseudo-supervised generative adversarial network and a medium. BACKGROUND

[0002] Traditional infrared and visible light image fusion (IVF) methods usually use linear or nonlinear transformation, filtering processing, image decomposition or sparse representation to extract image features, and use simple maximization, minimization, averaging or weighted summation to fuse image features. Although these methods are simple and efficient, their performance is limited due to the lack of guidance from real fused images. Modern deep learning methods usually use source input images as pseudo target images to guide the feature extraction and fusion process, but due to the lack of real fused images, these methods still have the problem of similarity balance. SUMMARY

[0003] The purpose of the embodiments of the present application is to provide an infrared and visible light image fusion method and system based on pseudo-supervised generative adversarial network and a medium, which introduces pseudo ground truth images and pseudo-supervised generative adversarial network to fuse and weight the images, and dynamically adjusts the network parameters, significantly improving the quality of the fused images.

[0004] The embodiments of the present application also provide an infrared and visible light image fusion method based on pseudo-supervised generative adversarial network, comprising:

[0005] Input the infrared image and the visible light image, and perform foreground and background segmentation on the infrared image and the visible light image based on the Otsu threshold method to obtain an infrared segmentation image and a visible light segmentation image;

[0006] Select a distance measurement method based on the infrared segmentation image and the visible light segmentation image to calculate the distance from the background pixel point to the nearest foreground pixel point, and generate an infrared distance image and a visible light distance image;

[0007] Perform weighting processing on the infrared distance image to obtain an infrared weighted image, perform weighting processing on the visible light distance image to obtain a visible light weighted image, and fuse the infrared weighted image and the visible light weighted image to obtain a pseudo ground truth image;

[0008] Perform convolution processing on the infrared image and the visible light image based on the pseudo-supervised generative adversarial network, analyze the convolution processing results to obtain a fused image, and perform enhancement processing on the fused image to obtain an enhanced fused image;

[0009] Compare and analyze the enhanced fused image and the pseudo ground truth image to obtain a discrimination result, and adjust the parameters of the pseudo-supervised generative adversarial network based on the discrimination result.

[0010] Optionally, in the infrared and visible image fusion method based on the pseudo-supervised generative adversarial network, the infrared image and the visible image are input, and the Otsu threshold method is used to perform foreground and background segmentation on the infrared image and the visible image respectively to obtain an infrared segmentation image and a visible segmentation image, specifically including:

[0011] An infrared image and a visible image are obtained, the number of pixels of each gray level is analyzed, and a gray histogram corresponding to the infrared image and a gray histogram corresponding to the visible image are generated;

[0012] The inter-class variances of different gray levels are calculated based on the gray histogram corresponding to the infrared image to generate a first segmentation threshold;

[0013] The infrared image is segmented based on the first segmentation threshold to obtain an infrared segmentation image;

[0014] The inter-class variances of different gray levels are calculated based on the gray histogram corresponding to the visible image to generate a second segmentation threshold;

[0015] The visible image is segmented based on the second segmentation threshold to obtain a visible segmentation image.

[0016] Optionally, in the infrared and visible image fusion method based on the pseudo-supervised generative adversarial network, the distance between the background pixel points and the nearest foreground pixel points is calculated based on the infrared segmentation image and the visible segmentation image to generate an infrared distance image and a visible distance image, specifically including:

[0017] The distance measurement method is selected based on the fusion target, and the distance measurement method includes Euclidean distance, Manhattan distance or Chebyshev distance;

[0018] The infrared segmentation image and the visible segmentation image are traversed in the order from left to right and from top to bottom

[0019] Based on each background pixel point, the distance value of each background pixel point to the nearest foreground pixel point is calculated;

[0020] After traversing the image once, the distance value of each pixel point is updated by reverse traversal from right to left and from bottom to top to generate an infrared distance image and a visible distance image.

[0021] Optionally, in the infrared and visible image fusion method based on the pseudo-supervised generative adversarial network, the infrared weighted image and the visible weighted image are fused to obtain a pseudo-ground truth image, specifically including:

[0022] The weight coefficient is obtained by weight distribution according to the distance value of each background pixel point to the nearest foreground pixel point.

[0023] weighting the infrared image and the visible light image based on the weight coefficient to obtain an infrared weighted image and a visible light weighted image;

[0024] calculating pixel values of the infrared weighted image and the visible light weighted image, and performing modal average fusion processing on the pixel values to obtain a pseudo-ground true image.

[0025] Optionally, in the infrared and visible light image fusion method based on the pseudo-supervised generative adversarial network, the fusion image is subjected to enhancement processing to obtain an enhanced fusion image, and the enhancement processing specifically includes:

[0026] performing Gaussian filtering processing on the fusion image to obtain a filtered image;

[0027] traversing each pixel of the filtered image, calculating a new value of each pixel according to a logarithmic transformation formula to obtain a transformed pixel value;

[0028] performing normalization processing on the transformed pixel value to map the pixel value to a set interval range;

[0029] performing subtraction operation on the fusion image after the normalization processing to obtain the enhanced fusion image.

[0030] Optionally, in the infrared and visible light image fusion method based on the pseudo-supervised generative adversarial network, the enhanced fusion image and the pseudo-ground true image are subjected to comparative analysis to obtain a discrimination result, and the parameters of the pseudo-supervised generative adversarial network are adjusted based on the discrimination result, and the comparative analysis specifically includes:

[0031] performing comparative analysis on the enhanced fusion image and the pseudo-ground true image to calculate a loss value of the pseudo-supervised generative adversarial network;

[0032] comparing the loss value with a set loss threshold value;

[0033] if the loss value is greater than the set loss threshold value, a correction coefficient is generated, and the network parameters of the pseudo-supervised generative adversarial network are adjusted based on the correction coefficient;

[0034] if the loss value is less than or equal to the set loss threshold value, image fusion information is generated.

[0035] In a second aspect, the embodiments of the present application provide an infrared and visible light image fusion system based on a pseudo-supervised generative adversarial network, which includes a memory and a processor, the memory includes a program of an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network, and the program of the infrared and visible light image fusion method based on the pseudo-supervised generative adversarial network is implemented when the processor is executed to implement the following steps:

[0036] The input infrared image and the visible light image are segmented based on an Otsu threshold method to obtain an infrared segmentation image and a visible light segmentation image.

[0037] The distance between the background pixel and the nearest foreground pixel is calculated based on the infrared segmentation image and the visible light segmentation image to generate an infrared distance image and a visible light distance image.

[0038] The infrared distance image is weighted to obtain an infrared weighted image, the visible light distance image is weighted to obtain a visible light weighted image, and the infrared weighted image and the visible light weighted image are fused to obtain a pseudo-ground truth image.

[0039] The infrared image and the visible light image are respectively processed based on the pseudo-supervised generative adversarial network, the processed results are analyzed to obtain a fusion image, and the fusion image is enhanced to obtain an enhanced fusion image.

[0040] The enhanced fusion image and the pseudo-ground truth image are compared and analyzed to obtain a judgment result, and the parameters of the pseudo-supervised generative adversarial network are adjusted based on the judgment result.

[0041] Optionally, in the infrared and visible light image fusion system based on the pseudo-supervised generative adversarial network, the input infrared image and the visible light image are segmented based on an Otsu threshold method to obtain an infrared segmentation image and a visible light segmentation image, and the method specifically comprises the following steps.

[0042] The infrared image and the visible light image are obtained, the number of pixels of each gray level is analyzed, and a gray histogram corresponding to the infrared image and a gray histogram corresponding to the visible light image are generated.

[0043] The inter-class variances of different gray levels are calculated based on the gray histogram corresponding to the infrared image to generate a first segmentation threshold.

[0044] The infrared image is segmented based on the first segmentation threshold to obtain an infrared segmentation image.

[0045] The inter-class variances of different gray levels are calculated based on the gray histogram corresponding to the visible light image to generate a second segmentation threshold.

[0046] The visible light image is segmented based on the second segmentation threshold to obtain a visible light segmentation image.

[0047] Optionally, in the infrared and visible image fusion system based on the pseudo-supervised generative adversarial network, the distance between the background pixel point and the nearest foreground pixel point is calculated based on the infrared segmentation image and the visible light segmentation image, and the infrared distance image and the visible light distance image are generated, and the method specifically comprises the following steps:

[0048] The distance metric method is selected based on the fusion target, and the distance metric method comprises Euclidean distance, Manhattan distance or Chebyshev distance.

[0049] The infrared segmentation image and the visible light segmentation image are traversed in the order from left to right and from top to bottom

[0050] Based on each background pixel point, the distance value from each background pixel point to the nearest foreground pixel point is calculated.

[0051] After traversing the image once, the distance value of each pixel point is updated through reverse traversal from right to left and from bottom to top, and the infrared distance image and the visible light distance image are generated.

[0052] In a third aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium comprises a pseudo-supervised generative adversarial network-based infrared and visible image fusion method program, and the pseudo-supervised generative adversarial network-based infrared and visible image fusion method program is executed by a processor to implement the steps of the pseudo-supervised generative adversarial network-based infrared and visible image fusion method according to any one of the above embodiments.

[0053] As can be seen from the above, the pseudo-supervised generative adversarial network-based infrared and visible image fusion method, system and medium provided by the embodiments of the present application input the infrared image and the visible light image, perform foreground and background segmentation on the infrared image and the visible light image based on the Otsu threshold method to obtain the infrared segmentation image and the visible light segmentation image, calculate the distance from the background pixel point to the nearest foreground pixel point based on the infrared segmentation image and the visible light segmentation image, generate the infrared distance image and the visible light distance image, perform weighting processing on the infrared distance image to obtain the infrared weighted image, perform weighting processing on the visible light distance image to obtain the visible light weighted image, fuse the infrared weighted image and the visible light weighted image to obtain the pseudo-ground truth image, perform convolution processing on the infrared image and the visible light image based on the pseudo-supervised generative adversarial network, analyze the convolution processing result to obtain the fusion image, perform enhancement processing on the fusion image to obtain the enhanced fusion image, compare and analyze the enhanced fusion image and the pseudo-ground truth image to obtain the discrimination result, adjust the parameters of the pseudo-supervised generative adversarial network based on the discrimination result, introduce the pseudo-ground truth image and the pseudo-supervised generative adversarial network to perform image fusion, and dynamically adjust the network parameters, thereby significantly improving the quality of the fusion image. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0055] Figure 1 Flowchart of the infrared and visible light image fusion method based on pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0056] Figure 2 A flowchart of an image segmentation method for an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0057] Figure 3 Flowchart of a pseudo ground truth image (PGI image) generation method for an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of the pseudo-supervised generative adversarial network structure of the infrared and visible light image fusion system based on the pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of the generator network structure of the infrared and visible light image fusion system based on the pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0060] Figure 6 Schematic diagram of the SSR component structure of the infrared and visible light image fusion system based on pseudo-supervised generative adversarial network provided in an embodiment of the present application;

[0061] Figure 7 Schematic diagram of the discriminator network structure of the infrared and visible light image fusion system based on the pseudo-supervised generative adversarial network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0063] It should be noted that similar reference numbers and letters represent similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0064] Please refer to Figure 1 , Figure 1 is a flowchart of an infrared and visible image fusion method based on a pseudo-supervised generative adversarial network in some embodiments of the present application. The infrared and visible image fusion method based on the pseudo-supervised generative adversarial network is used in a terminal device. The infrared and visible image fusion method based on the pseudo-supervised generative adversarial network comprises the following steps:

[0065] S101, inputting an infrared image and a visible light image, performing foreground and background segmentation on the infrared image and the visible light image based on an Otsu threshold method to obtain an infrared segmentation image and a visible light segmentation image;

[0066] S102, selecting a distance measurement method to calculate the distance from a background pixel point to the nearest foreground pixel point based on the infrared segmentation image and the visible light segmentation image, generating an infrared distance image and a visible light distance image;

[0067] S103, performing weighted processing on the infrared distance image to obtain an infrared weighted image, performing weighted processing on the visible light distance image to obtain a visible light weighted image, and fusing the infrared weighted image and the visible light weighted image to obtain a pseudo-ground truth image;

[0068] S104, performing convolution processing on the infrared image and the visible light image based on a pseudo-supervised generative adversarial network, analyzing the convolution processing result to obtain a fusion image, and performing enhancement processing on the fusion image to obtain an enhanced fusion image;

[0069] S105, compare the enhanced fusion image with the pseudo-ground true image, obtain a discrimination result, and adjust the parameters of the pseudo-supervised generative adversarial network based on the discrimination result.

[0070] It should be noted that the calculation process of the pseudo-ground true image (PGI image) includes the following steps:

[0071] (1) Image segmentation: using Otsu threshold method to segment the input image (infrared image and visible light image) I in into foreground and background, to obtain a binary segmentation image I sb .

[0072] (2) Distance image generation: calculate the distance image I dm , the formula is as follows:

[0073]

[0074] Where I sb (m,n) = 0 represents that the pixel (m,n) is a background pixel, and I dm (i,j) represents the distance from the pixel (i,j) to the nearest background pixel.

[0075] (3) Distance weighted image generation: calculate the normalized distance weight image I dw , the formula is as follows:

[0076]

[0077] I dw represents the normalized distance weighted image (final output), that is, the image obtained by multiplying the original image with the weight.

[0078] I dm represents the distance map (Distance Map), that is, the distance from each pixel to the nearest foreground pixel.

[0079] M represents the number of rows (height) of the image; n represents the number of columns (width) of the image, that is, the size of the image.

[0080] C (i,j) represents the set of all pixels in the connected region where the pixel (i,j) is located, which is used for normalization.

[0081] i,j respectively represent the row and column coordinate positions of the pixels in the image.

[0082] otherwise represents the processing logic when the main condition is not met.

[0083] Where C (i,j) represents the connected region where the pixel (i,j) is located.

[0084] (4) Residual Sum: calculate the Residual Sum Image I according to the formula rs :

[0085] I rs dw1 in dw in in

[0086] I rs denotes the Residual Sum Image, which is used to reflect the difference between the infrared / visible light modal weighted image and the original image.

[0087] I dw1 denotes the distance-weighted infrared image, which is obtained by weighting the infrared image and its distance map.

[0088] I in denotes the original input image;

[0089] I dw denotes the image obtained by multiplying the original image (infrared image or visible light image) with its corresponding distance map according to a certain weight coefficient.

[0090] Specifically, the residual refers to the difference between the predicted value and the true value of the model. In image fusion, residual sum usually refers to first performing certain operations (such as prediction, transformation, etc.) on images of different modalities to obtain respective results, then calculating the residual between these results and the original images, and finally summing the residuals. For example, a generator is used to generate a fusion prediction image for the infrared image and the visible light image, the generated image is subtracted from the original infrared and visible light images to obtain residual images, and the residual values of corresponding pixels of the two residual images are added. Through residual sum, the parts of different modal images that are not accurately represented after processing can be captured, which helps to further optimize the fusion process or model.

[0091] (5) Modal Average: calculate the PGI image according to the formula

[0092]

[0093] I denotes the image obtained by distance weighting (residual-weighted) processing of the infrared image, i.e., the residual-weighted image of the infrared modality.

[0094] ​​​​​The visible light image after distance weighting processing, that is, the residual weighted image of the visible light modality.

[0095] Specifically, modal average is for multi-modal data (such as infrared and visible light images), and the images or features of different modalities are averaged. Specifically, the corresponding pixel values (or feature values) of the infrared image and the visible light image are added, and then divided by the number of modalities (here, 2).

[0096] By introducing the pseudo ground truth image (PGI) and the single-scale Retinex (SSR) embedding, the problem of lack of real fusion images in the prior art is solved, and the quality of the fusion image is improved.

[0097] Please refer to Figure 2 , Figure 2 is a flow chart of an image segmentation method of an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network in some embodiments of the present application. According to the embodiment of the present application, the infrared image and the visible light image are input, and the Otsu threshold method is used to segment the foreground and background of the infrared image and the visible light image respectively to obtain the infrared segmentation image and the visible light segmentation image, specifically including:

[0098] S201, obtaining the infrared image and the visible light image, analyzing the number of pixels of each gray level, and generating a gray level histogram corresponding to the infrared image and a gray level histogram corresponding to the visible light image;

[0099] S202, calculating the inter-class variance of different gray levels based on the gray level histogram corresponding to the infrared image to generate a first segmentation threshold;

[0100] S203, segmenting the foreground and background of the infrared image based on the first segmentation threshold to obtain the infrared segmentation image;

[0101] S204, calculating the inter-class variance of different gray levels based on the gray level histogram corresponding to the visible light image to generate a second segmentation threshold;

[0102] S205, segmenting the foreground and background of the visible light image based on the second segmentation threshold to obtain the visible light segmentation image.

[0103] It should be noted that the Otsu threshold method, also known as the Otsu method, is an unsupervised threshold segmentation algorithm based on the statistical characteristics of image gray levels. By calculating the inter-class variance of different gray levels in the image, an optimal threshold is found, which makes the inter-class variance reach the maximum after the image is segmented into foreground and background. The larger the inter-class variance, the more obvious the difference between the foreground and the background, and the better the segmentation effect.

[0104] According to the embodiment of the present application, the distance from the background pixel point to the nearest foreground pixel point is calculated based on the infrared segmentation image and the visible light segmentation image to generate the infrared distance image and the visible light distance image, and the specific steps include:

[0105] The distance measurement method is selected based on the fusion target, and the distance measurement method includes Euclidean distance, Manhattan distance or Chebyshev distance.

[0106] The infrared segmentation image and the visible light segmentation image are traversed in the order from left to right and from top to bottom

[0107] Based on each background pixel point, the distance value from each background pixel point to the nearest foreground pixel point is calculated.

[0108] After traversing the image once, reverse traversal is performed from right to left and from bottom to top to update the distance value of each pixel point, and the infrared distance image and the visible light distance image are generated.

[0109] It should be noted that the Euclidean distance, Manhattan distance and Chebyshev distance are all suitable for numerical data. If the data is in a continuous space and meets the characteristics of Euclidean space, the Euclidean distance is usually a natural choice; if the data has a grid or discrete characteristic, the Manhattan distance may be more suitable; and when the maximum difference of the data in each dimension is concerned, the Chebyshev distance is more suitable.

[0110] Please refer to Figure 3 , Figure 3 is a pseudo ground truth image (PGI image) generation method flowchart of an infrared and visible light image fusion method based on pseudo-supervised generative adversarial network in some embodiments of the present application. According to the embodiment of the present application, the infrared weighted image and the visible light weighted image are fused to obtain the pseudo ground truth image, and the specific steps include:

[0111] The weight distribution is performed according to the distance value from each background pixel point to the nearest foreground pixel point to obtain the weight coefficient.

[0112] The infrared image and the visible light image are weighted based on the weight coefficient to obtain the infrared weighted image and the visible light weighted image.

[0113] The pixel values of the infrared weighted image and the visible light weighted image are calculated, and the pixel values are subjected to modal average fusion processing to obtain the pseudo ground truth image.

[0114] It should be noted that distance weighting is a method of assigning weights according to the distance between elements. In the context of image fusion, such as when processing features or pixels of different modal images, the distance of each element (pixel or feature point) from a certain reference point or region is calculated. Elements with shorter distances are assigned higher weights, and elements with longer distances are assigned lower weights. For example, for infrared and visible light image fusion, when fusing features, the distance of each pixel in the visible light image to a certain target region in the infrared image can be calculated, and pixels closer to the target region have greater weights in the fusion process.

[0115] According to an embodiment of the present application, the fused image is subjected to enhancement processing to obtain an enhanced fused image, specifically including:

[0116] The fused image is subjected to Gaussian filtering processing to obtain a filtered image;

[0117] Each pixel of the filtered image is traversed, and the new value of each pixel is calculated according to a logarithmic transformation formula to obtain a transformed pixel value;

[0118] The transformed pixel value is subjected to normalization processing to map the pixel value to a set interval range;

[0119] The normalized fused image is subjected to subtraction operation to obtain the enhanced fused image.

[0120] It should be noted that each pixel of the image is traversed, and the new value of each pixel is calculated according to the logarithmic transformation formula. If the image is a color image, the logarithmic transformation needs to be performed on each color channel (such as RGB channel) respectively.

[0121] Normalization processing: Since the pixel value after logarithmic transformation may not be within the pixel value range of the original image (such as 0-255), normalization processing needs to be performed on the result to map the pixel value back to a suitable range. To some extent, the logarithmic operation can suppress noise in the image.

[0122] Display or save the image: the processed image is displayed or saved for viewing the effect after the logarithmic operation.

[0123] According to an embodiment of the present application, the enhanced fused image is compared and analyzed with the pseudo-ground-truth image to obtain a judgment result, and the parameters of the pseudo-supervised generative adversarial network are adjusted based on the judgment result, specifically including:

[0124] The enhanced fused image is compared and analyzed with the pseudo-ground-truth image to calculate the loss value of the pseudo-supervised generative adversarial network;

[0125] The loss value is compared with a set loss threshold value;

[0126] If the loss value is greater than the set loss threshold, a correction coefficient is generated, and the network parameters of the pseudo-supervised generative adversarial network are adjusted based on the correction coefficient;

[0127] If the loss value is less than or equal to the set loss threshold, image fusion information is generated.

[0128] It should be noted that the loss value of the pseudo-supervised generative adversarial network includes a generator loss part and a discriminator loss part, specifically,

[0129] Generator loss: The goal of the generator is to generate a fusion image that can deceive the discriminator. Its loss function usually includes an adversarial loss and a pseudo-supervised loss. The adversarial loss uses a cross-entropy loss or other similar loss function to measure the probability that the fusion image generated by the generator is judged as a real image by the discriminator. The pseudo-supervised loss is defined according to specific pseudo-supervised information, such as using a cross-entropy loss to measure the difference between the class prediction of the generated image and the pseudo-label.

[0130] Discriminator loss: The goal of the discriminator is to accurately distinguish between real fusion images and generated fusion images. Its loss function also uses a cross-entropy loss to maximize the probability of correct classification. At the same time, combined with pseudo-supervised information, the ability to distinguish image semantic information is increased.

[0131] According to an embodiment of the present application, the pseudo-supervised generative adversarial network (PS-GAN network) structure includes a generator, an SSR component, and a discriminator.

[0132] SSR is an image enhancement algorithm designed to simulate the perception of the human visual system for images under different lighting conditions, and is particularly suitable for improving image contrast, enhancing details, and highlighting textures.

[0133] SSR is embedded in the pseudo-supervised generative adversarial network (PS-GAN) to process the fusion image and improve its visual quality, making it closer to the "pseudo ground truth image" (PGI).

[0134] The generator uses a double-encoder-single-decoder structure, and the encoders E IR and E VIS process infrared and visible light images, respectively. The decoder concatenates the outputs of the two encoders and performs image reconstruction.

[0135] The SSR component is embedded between the generator and the discriminator to enhance the contrast of the fusion image. The output I ef According to the formula:

[0136]

[0137] I efEnhanced Fused Image, refers to the image output by the fusion network after enhancement processing (such as Gaussian filter, logarithmic transformation, normalization, etc.). It is used for comparison with PGI image as one of the inputs of the discriminator.

[0138] I fused Fused Image refers to the initial fusion image directly output by the generator of the pseudo-supervised generative adversarial network (PS-GAN), which has not been enhanced.

[0139] where F is a Gaussian filter, and the formula is as follows:

[0140]

[0141] σ represents the standard deviation of the Gaussian filter, which controls the blurring degree. F(x,y) represents the standard deviation of the Gaussian filter, which controls the blurring degree;

[0142] x,y represents the spatial coordinates (consistent with the image pixel coordinates).

[0143] The discriminator is used to distinguish PGI and enhanced fused image I ef The discriminator is usually a convolutional neural network (CNN). It is composed of multiple convolutional layers, pooling layers and fully connected layers. Convolutional layers are used to extract features of images, pooling layers are used to reduce the dimension of feature maps, and fully connected layers are used to map extracted features to a binary classification result, i.e. to judge whether the input image is a real fused image or a false fused image generated by the generator.

[0144] Function: The core function of the discriminator is to distinguish between real fused images and fused images generated by the generator. It learns the feature distribution of real fused images to judge whether the input image conforms to the feature pattern of real images. In the training process, the discriminator continuously adjusts its parameters to improve the ability to distinguish between real images and generated images, and provides feedback to the generator to guide the generator to generate fused images closer to real images.

[0145] The loss function of the PS-GAN network includes generator loss and discriminator loss.

[0146] Discriminator loss: the discriminator loss L D is calculated according to the formula:

[0147]

[0148] where b=1, a=0; D()∈(0,1) represents the possibility that the input image is a real fused image (PGI). n represents the sample index, i.e. the nth image sample. N represents the total number of samples, i.e. the number of images in a batch.

[0149] Generator loss: Generator loss L G Including class mean square loss L CMS and content loss L content , the formula is as follows:

[0150] L G =L cms +γ·L content

[0151] Among them, γ is a hyperparameter, content loss L content Calculated according to the formula:

[0152] L content =L ss +αL gradient

[0153] Among them, L ss and L gradient They represent the structural similarity loss and gradient similarity loss respectively; α represents the weight coefficient of the structural similarity loss.

[0154] like Figures 4-7 As shown, in a second aspect, an embodiment of the present application provides an infrared and visible light image fusion system based on a pseudo-supervised generative adversarial network, the system comprising: a memory and a processor, the memory comprising a program of an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network, and when the program of the infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network is executed by the processor, the following steps are implemented:

[0155] Input infrared image and visible light image, perform foreground and background segmentation on the infrared image and visible light image respectively based on Otsu threshold method, and obtain infrared segmentation image and visible light segmentation image;

[0156] Based on the infrared segmentation image and the visible light segmentation image, a distance measurement method is selected to calculate the distance from the background pixel to the nearest foreground pixel, and an infrared range image and a visible light range image are generated;

[0157] The infrared range image is weighted to obtain an infrared weighted image, the visible light range image is weighted to obtain a visible light weighted image, and the infrared weighted image and the visible light weighted image are fused to obtain a pseudo ground truth image;

[0158] Based on the pseudo-supervised generative adversarial network, the infrared image and the visible light image are convolved respectively, the convolution processing results are analyzed to obtain a fused image, and the fused image is enhanced to obtain an enhanced fused image;

[0159] The enhanced fusion image is compared with the pseudo-ground real image to obtain a discrimination result, and parameters of the pseudo-supervised generative adversarial network are adjusted based on the discrimination result.

[0160] According to the embodiment of the present application, the infrared image and the visible light image are input, and the Otsu threshold method is used to perform foreground and background segmentation on the infrared image and the visible light image respectively to obtain an infrared segmentation image and a visible light segmentation image, specifically including:

[0161] The infrared image and the visible light image are acquired, the number of pixels of each gray level is analyzed, and a gray histogram corresponding to the infrared image and a gray histogram corresponding to the visible light image are generated;

[0162] The inter-class variance of different gray levels is calculated based on the gray histogram corresponding to the infrared image to generate a first segmentation threshold;

[0163] The infrared image is segmented into foreground and background based on the first segmentation threshold to obtain an infrared segmentation image;

[0164] The inter-class variance of different gray levels is calculated based on the gray histogram corresponding to the visible light image to generate a second segmentation threshold;

[0165] The visible light image is segmented into foreground and background based on the second segmentation threshold to obtain a visible light segmentation image.

[0166] It should be noted that the Otsu threshold method, also known as the Otsu method, is an unsupervised threshold segmentation algorithm based on the statistical characteristics of image gray levels, which finds an optimal threshold by calculating the inter-class variance of different gray levels in the image, so that the inter-class variance reaches the maximum after the image is segmented into foreground and background. The greater the inter-class variance, the more obvious the difference between the foreground and the background, and the better the segmentation effect.

[0167] According to the embodiment of the present application, the distance from each background pixel point to the nearest foreground pixel point is calculated based on the infrared segmentation image and the visible light segmentation image to generate an infrared distance image and a visible light distance image, specifically including:

[0168] The distance measurement method is selected based on the fusion target, and the distance measurement method includes Euclidean distance, Manhattan distance or Chebyshev distance;

[0169] The infrared segmentation image and the visible light segmentation image are traversed in the order from left to right and from top to bottom

[0170] Based on each background pixel point, the distance value from each background pixel point to the nearest foreground pixel point is calculated;

[0171] After traversing the image once, the distance values of each pixel point are updated from right to left and from bottom to top in reverse traversal to generate the infrared distance image and the visible light distance image.

[0172] It should be noted that the Euclidean distance, Manhattan distance and Chebyshev distance are all applicable to numerical data. If the data is in a continuous space and meets the characteristics of Euclidean space, the Euclidean distance is usually a natural choice; if the data has a grid-like or discrete characteristic, the Manhattan distance may be more appropriate; and when the maximum difference of the data in each dimension is concerned, the Chebyshev distance is more suitable.

[0173] Please refer to Figure 3 , Figure 3 is a pseudo ground truth image (PGI image) generation method flowchart of an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network in some embodiments of the present application. According to the embodiment of the present application, the infrared weighted image and the visible light weighted image are fused to obtain the pseudo ground truth image, which specifically comprises:

[0174] According to the distance value of each background pixel point to the nearest foreground pixel point, a weight coefficient is obtained by weight distribution;

[0175] Based on the weight coefficient, the infrared image and the visible light image are weighted to obtain the infrared weighted image and the visible light weighted image;

[0176] The pixel values of the infrared weighted image and the visible light weighted image are calculated, and the pixel values are subjected to modal average fusion processing to obtain the pseudo ground truth image.

[0177] It should be noted that distance weighting is a method of assigning weights according to the distance between elements. In the context of image fusion, such as when processing features or pixels of different modal images, the distance of each element (pixel or feature point) to a specific reference point or region is calculated. Elements with a shorter distance are given a higher weight, and elements with a longer distance are given a lower weight. For example, for infrared and visible light image fusion, when fusing features, the distance of each pixel in the visible light image to a specific target region in the infrared image can be calculated, and the pixels closer to the target region have a greater weight in the fusion process.

[0178] According to the embodiment of the present application, the fused image is subjected to enhancement processing to obtain an enhanced fused image, which specifically comprises:

[0179] The fused image is subjected to Gaussian filtering processing to obtain a filtered image;

[0180] Each pixel of the filtered image is traversed, and the new value of each pixel is calculated according to a logarithmic transformation formula to obtain a transformed pixel value;

[0181] The transformed pixel value is normalized to map the pixel value to a set interval range.

[0182] The normalized fusion image is subjected to a subtraction operation to obtain an enhanced fusion image.

[0183] It should be noted that each pixel of the image is traversed, and the new value of each pixel is calculated according to the logarithmic transformation formula. If the image is a color image, the logarithmic transformation needs to be performed on each color channel (such as the RGB channel) respectively.

[0184] Normalization: Since the pixel value after logarithmic transformation may not be within the pixel value range of the original image (such as 0-255), normalization needs to be performed on the result to map the pixel value back to a suitable range. To some extent, the logarithmic operation can suppress the noise in the image.

[0185] Display or save the image: the processed image is displayed or saved for viewing the effect after the logarithmic operation.

[0186] According to the embodiment of the present application, the enhanced fusion image and the pseudo-ground-truth image are compared and analyzed to obtain a judgment result, and the parameters of the pseudo-supervised generative adversarial network are adjusted based on the judgment result, specifically including:

[0187] The enhanced fusion image and the pseudo-ground-truth image are compared and analyzed to calculate the loss value of the pseudo-supervised generative adversarial network;

[0188] The loss value is compared with a set loss threshold value;

[0189] If the loss value is greater than the set loss threshold value, a correction coefficient is generated, and the network parameters of the pseudo-supervised generative adversarial network are adjusted based on the correction coefficient;

[0190] If the loss value is less than or equal to the set loss threshold value, image fusion information is generated.

[0191] It should be noted that the loss value of the pseudo-supervised generative adversarial network includes a generator loss part and a discriminator loss part, specifically,

[0192] Generator loss: The goal of the generator is to generate a fusion image that can deceive the discriminator. Its loss function usually includes an adversarial loss and a pseudo-supervised loss. The adversarial loss uses a cross-entropy loss or other similar loss function to measure the probability that the fusion image generated by the generator is judged as a real image by the discriminator. The pseudo-supervised loss is defined according to specific pseudo-supervised information, such as using a cross-entropy loss to measure the difference between the class prediction of the generated image and the pseudo-label.

[0193] Discriminator loss: The goal of the discriminator is to accurately distinguish between real and generated fusion images. Its loss function also uses cross-entropy loss, maximizing the probability of correct classification. At the same time, combined with pseudo-supervised information, it increases the ability to distinguish the semantic information of the image.

[0194] According to an embodiment of the present application, the pseudo-supervised generative adversarial network (PS-GAN network) structure includes a generator, an SSR component and a discriminator.

[0195] The generator adopts a double-encoder-single-decoder structure, and the encoder E IR and E VIS process infrared and visible light images respectively. The decoder concatenates the outputs of the two encoders and then reconstructs the image.

[0196] The SSR component is embedded between the generator and the discriminator, and is used to enhance the contrast of the fusion image. The output I ef is calculated according to the formula:

[0197]

[0198] where F is a Gaussian filter, and the formula is as follows:

[0199]

[0200] The discriminator is used to distinguish between PGI and the enhanced fusion image I ef . The discriminator is usually a convolutional neural network (CNN). It is composed of multiple convolutional layers, pooling layers and fully connected layers. The convolutional layer is used to extract the features of the image, the pooling layer is used to reduce the dimension of the feature map, and the fully connected layer is used to map the extracted features to a binary classification result, that is, to judge whether the input image is a real fusion image or a false fusion image generated by the generator.

[0201] Function: The core function of the discriminator is to distinguish between real fusion images and fusion images generated by the generator. It learns the feature distribution of real fusion images to determine whether the input image conforms to the feature pattern of real images. In the training process, the discriminator continuously adjusts its parameters to improve the ability to distinguish between real images and generated images, and provides feedback to the generator to guide the generator to generate fusion images closer to real images.

[0202] The loss function of the PS-GAN network includes the generator loss and the discriminator loss.

[0203] Discriminator loss: The discriminator loss L D is calculated according to the formula:

[0204]

[0205] Wherein, b = 1, a = 0.

[0206] Generator loss: the generator loss L G Including class mean square loss L CMS And content loss L content , the formula is as follows:

[0207] L G = L cms + gamma L content

[0208] Wherein, gamma is a hyperparameter, content loss L content According to the formula:

[0209] L content = L ss + alpha L gradient

[0210] Wherein, L ss And L gradient Respectively represent the structural similarity loss and gradient similarity loss.

[0211] The third aspect of the application provides a computer readable storage medium, the readable storage medium includes the infrared and visible light image fusion method program based on the pseudo-supervised generative adversarial network, the infrared and visible light image fusion method program based on the pseudo-supervised generative adversarial network is executed by the processor, realizes the steps of the infrared and visible light image fusion method based on the pseudo-supervised generative adversarial network of any one of the above.

[0212] The application discloses an infrared and visible light image fusion method and system based on a pseudo-supervised generative adversarial network and a medium, which comprises the following steps: inputting an infrared image and a visible light image; performing foreground and background segmentation on the infrared image and the visible light image based on an Otsu threshold method to obtain an infrared segmentation image and a visible light segmentation image; selecting a distance measurement method to calculate the distance from a background pixel point to the nearest foreground pixel point based on the infrared segmentation image and the visible light segmentation image, and generating an infrared distance image and a visible light distance image; performing weighting processing on the infrared distance image to obtain an infrared weighted image, and performing weighting processing on the visible light distance image to obtain a visible light weighted image; fusing the infrared weighted image and the visible light weighted image to obtain a pseudo-ground truth image; performing convolution processing on the infrared image and the visible light image based on a pseudo-supervised generative adversarial network, analyzing the convolution processing result, obtaining a fusion image, and performing enhancement processing on the fusion image to obtain an enhanced fusion image; comparing and analyzing the enhanced fusion image and the pseudo-ground truth image to obtain a discrimination result, and adjusting the parameters of the pseudo-supervised generative adversarial network based on the discrimination result; introducing the pseudo-ground truth image and the pseudo-supervised generative adversarial network to perform weighted fusion on the image, and dynamically adjusting the network parameters, thereby significantly improving the quality of the fusion image.

[0213] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other manners. The above described device embodiments are merely exemplary. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0214] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place, or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0215] In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional units.

[0216] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by a program instructing related hardware, and the foregoing program can be stored in a readable storage medium, and when the program is executed, the steps of the above method embodiments are executed; and the foregoing storage medium includes: mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic discs or optical discs, and various media that can store program codes.

[0217] Or, when the integrated units of the present application are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the methods described in the embodiments of the present application. The foregoing storage medium includes: mobile storage devices, ROMs, RAMs, magnetic discs or optical discs, and various media that can store program codes.

Claims

1. A method for fusion of infrared and visible light images based on pseudo-supervised generative adversarial networks, characterized in that: include: Input infrared image and visible light image, perform foreground and background segmentation on the infrared image and visible light image respectively based on Otsu threshold method, and obtain infrared segmentation image and visible light segmentation image; Based on the infrared segmentation image and the visible light segmentation image, a distance measurement method is selected to calculate the distance from the background pixel to the nearest foreground pixel, and an infrared range image and a visible light range image are generated; The infrared range image is weighted to obtain an infrared weighted image, the visible light range image is weighted to obtain a visible light weighted image, and the infrared weighted image and the visible light weighted image are fused to obtain a pseudo ground truth image; Based on the pseudo-supervised generative adversarial network, the infrared image and the visible light image are convolved respectively, the convolution processing results are analyzed to obtain a fused image, and the fused image is enhanced to obtain an enhanced fused image; The enhanced fusion image is compared and analyzed with the pseudo ground truth image to obtain the discrimination result, and the parameters of the pseudo supervised generative adversarial network are adjusted based on the judgment result.

2. The infrared and visible light image fusion method based on pseudo-supervised generative adversarial network according to claim 1 is characterized in that: Input infrared image and visible light image, perform foreground and background segmentation on the infrared image and visible light image respectively based on Otsu threshold method, and obtain infrared segmentation image and visible light segmentation image, specifically including: Obtain an infrared image and a visible light image, analyze the number of pixels that appear at each gray level, and generate a grayscale histogram corresponding to the infrared image and a grayscale histogram corresponding to the visible light image; Calculating the inter-class variance of different gray levels based on the grayscale histogram corresponding to the infrared image to generate a first segmentation threshold; Segmenting the infrared image into foreground and background based on a first segmentation threshold to obtain an infrared segmented image; Calculating the inter-class variance of different gray levels based on the grayscale histogram corresponding to the visible light image to generate a second segmentation threshold; The visible light image is segmented into foreground and background based on the second segmentation threshold to obtain a visible light segmented image.

3. The infrared and visible light image fusion method based on pseudo-supervised generative adversarial network according to claim 2 is characterized in that: Based on the infrared segmentation image and the visible light segmentation image, a distance measurement method is selected to calculate the distance from the background pixel to the nearest foreground pixel, and an infrared range image and a visible light range image are generated, specifically including: Selecting a distance metric based on the fusion target, wherein the distance metric includes Euclidean distance, Manhattan distance, or Chebyshev distance; Traverse the infrared segmentation image and the visible light segmentation image from left to right and from top to bottom, and calculate the distance value from each background pixel to the nearest foreground pixel based on each background pixel; After traversing the image once, reverse traversal is performed from right to left and from bottom to top to update the distance value of each pixel to generate an infrared range image and a visible light range image.

4. The infrared and visible light image fusion method based on pseudo-supervised generative adversarial network according to claim 3 is characterized in that: The infrared weighted image is fused with the visible light weighted image to obtain the pseudo ground truth image, which specifically includes: The weight coefficient is obtained by weighting the distance value from each background pixel to the nearest foreground pixel. The infrared image and the visible light image are weighted based on the weight coefficient to obtain an infrared weighted image and a visible light weighted image; The pixel values ​​of the infrared weighted image and the visible light weighted image are calculated, and the pixel values ​​are fused by modal averaging to obtain the pseudo ground truth image.

5. The infrared and visible light image fusion method based on pseudo-supervised generative adversarial network according to claim 4 is characterized in that: The fused image is enhanced to obtain an enhanced fused image, specifically including: Perform Gaussian filtering on the fused image to obtain a filtered image; Traverse each pixel of the filtered image, calculate the new value of each pixel according to the logarithmic transformation formula, and obtain the transformed pixel value; Normalize the transformed pixel values ​​and map the pixel values ​​to a set interval range; The normalized fused image is subtracted to obtain an enhanced fused image.

6. The infrared and visible light image fusion method based on pseudo-supervised generative adversarial network according to claim 5 is characterized in that: The enhanced fusion image is compared with the pseudo ground truth image to obtain the discrimination result. Based on the discrimination result, the parameters of the pseudo-supervised generative adversarial network are adjusted, including: Compare and analyze the enhanced fusion image with the pseudo ground truth image and calculate the loss value of the pseudo-supervised generative adversarial network; Compare the loss value with the set loss threshold; If the loss value is greater than the set loss threshold, a correction coefficient is generated, and the network parameters of the pseudo-supervised generative adversarial network are adjusted based on the correction coefficient; If the loss value is less than or equal to the set loss threshold, image fusion information is generated.

7. An infrared and visible light image fusion system based on pseudo-supervised generative adversarial network, characterized by: The system includes: a memory and a processor, wherein the memory includes a program of an infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network, and when the program of the infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network is executed by the processor, the following steps are implemented: Input infrared image and visible light image, perform foreground and background segmentation on the infrared image and visible light image respectively based on Otsu threshold method, and obtain infrared segmentation image and visible light segmentation image; Based on the infrared segmentation image and the visible light segmentation image, a distance measurement method is selected to calculate the distance from the background pixel to the nearest foreground pixel, and an infrared range image and a visible light range image are generated; The infrared range image is weighted to obtain an infrared weighted image, the visible light range image is weighted to obtain a visible light weighted image, and the infrared weighted image and the visible light weighted image are fused to obtain a pseudo ground truth image; Based on the pseudo-supervised generative adversarial network, the infrared image and the visible light image are convolved respectively, the convolution processing results are analyzed to obtain a fused image, and the fused image is enhanced to obtain an enhanced fused image; The enhanced fusion image is compared and analyzed with the pseudo ground truth image to obtain the discrimination result, and the parameters of the pseudo supervised generative adversarial network are adjusted based on the judgment result.

8. The infrared and visible light image fusion system based on pseudo-supervised generative adversarial network according to claim 7, characterized in that: Input infrared image and visible light image, perform foreground and background segmentation on the infrared image and visible light image respectively based on Otsu threshold method, and obtain infrared segmentation image and visible light segmentation image, specifically including: Obtain an infrared image and a visible light image, analyze the number of pixels that appear at each gray level, and generate a grayscale histogram corresponding to the infrared image and a grayscale histogram corresponding to the visible light image; Calculating the inter-class variance of different gray levels based on the grayscale histogram corresponding to the infrared image to generate a first segmentation threshold; Segmenting the infrared image into foreground and background based on a first segmentation threshold to obtain an infrared segmented image; Calculating the inter-class variance of different gray levels based on the grayscale histogram corresponding to the visible light image to generate a second segmentation threshold; The visible light image is segmented into foreground and background based on the second segmentation threshold to obtain a visible light segmented image.

9. The infrared and visible light image fusion system based on pseudo-supervised generative adversarial network according to claim 8, characterized in that: Based on the infrared segmentation image and the visible light segmentation image, a distance measurement method is selected to calculate the distance from the background pixel to the nearest foreground pixel, and an infrared range image and a visible light range image are generated, specifically including: Selecting a distance metric based on the fusion target, wherein the distance metric includes Euclidean distance, Manhattan distance, or Chebyshev distance; Traverse the infrared segmentation image and the visible light segmentation image from left to right and from top to bottom, and calculate the distance value from each background pixel to the nearest foreground pixel based on each background pixel; After traversing the image once, reverse traversal is performed from right to left and from bottom to top to update the distance value of each pixel to generate an infrared range image and a visible light range image.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a program for the infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network. When the program for the infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network is executed by a processor, the steps of the infrared and visible light image fusion method based on a pseudo-supervised generative adversarial network as described in any one of claims 1 to 6 are implemented.

Images