Infrared and visible light image fusion method based on improved StyleGAN2

Abstract

The invention relates to the field of image processing, in particular to an infrared and visible light image fusion method based on improved StyleGAN2, which comprises the following steps: constructing a generator of the improved StyleGAN2, the generator comprises a style encoder and a content encoder, the style encoder is used for outputting a style vector of an image, the content encoder is used for outputting a content feature of the image, and the content encoder is used for outputting a content feature of the image; the generator is used for generating a fused image; constructing a discriminator for judging the similarity between the fused image and the infrared image and the similarity between the fused image and the visible light image; and the optimal improved StyleGAN2 is obtained by using the loss function and is used for fusion of the infrared image and the visible light image. According to the multi-modal feature fusion method, more balanced multi-modal feature fusion is realized through a dual-discriminator fusion framework and collaborative optimization of feature discrimination capabilities of two modals.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an infrared and visible light image fusion method based on improved StyleGAN2. BACKGROUND

[0002] Visible and infrared image fusion (VIF) provides a comprehensive scene description by combining infrared thermal images with the rich texture of visible light cameras, and has a wide application prospect in the fields of robot vision, monitoring and remote sensing. Visible and infrared image fusion methods can be roughly divided into three categories: traditional image fusion methods based on spatial domain and transform domain, fusion methods based on convolutional neural networks, and deep fusion methods based on generative adversarial networks (GAN).

[0003] Traditional image fusion requires manual design of fusion strategies, and the fusion performance is limited. Moreover, the same transformation is often used to extract features from different source images, without considering the differences in source image features, which may result in poor feature expression ability. Deep learning-based methods can design different branches or modules to achieve differentiated feature extraction and obtain more targeted features. In addition, the use of a well-designed loss function can also learn a more reasonable feature fusion strategy.

[0004] Compared with traditional image fusion methods, GAN provides a new training mode, and the parameter update of the generator relies on the discriminator in a certain sense. A large amount of existing data shows that GAN can generate clear and realistic samples. Researchers have proposed various image fusion methods based on generative adversarial networks, which avoid the design of complex fusion rules by utilizing the end-to-end characteristics of generative adversarial networks. FusionGAN describes the fusion task of infrared and visible light images as an adversarial game between preserving infrared thermal radiation information and preserving visible appearance texture information, i.e., the generator attempts to generate a fusion image with primary infrared intensity and additional visible gradient, while the discriminator aims to force the fusion image to have more texture details. This enables the fusion image to simultaneously preserve the thermal radiation in the infrared image and the texture details in the visible light image.

[0005] However, the research on infrared and visible light image fusion based on GAN still faces many challenges: (1) Most GAN-based image fusion methods use only one discriminator. In the discrimination process, although more visible light image contour texture details can be fitted, the problem of less infrared image information preservation may occur.

[0006] (2) In many current end-to-end GAN image fusion methods, the generator mostly has a simple network structure, resulting in insufficient and unbalanced feature extraction of infrared and visible light images, and the problem of ignoring small detail features.

[0007] The existing GAN-based fusion image generation method has FusionGAN. The architecture design causes the generator to tend to generate images closer to the visible light features to better deceive the discriminator. Therefore, the "fusion image" finally output by the model is actually more similar to the grayscale version of the visible light image, failing to sufficiently retain the key thermal radiation features of the infrared image. To solve this problem, GANMcC introduces infrared and visible light image discrimination tasks in the discriminator. Specifically, the discriminator needs to evaluate the similarity of the fusion image with the two source images respectively, and the optimization goal of the generator is to maximize the two similarity probabilities. This multi-distribution joint estimation framework, through an antagonistic game, causes the fusion result to more evenly retain the features of the two modalities. However, GANMcC still has the following limitations. First, the generator network is relatively simple and has limited feature extraction capability. Second, the single discriminator architecture easily leads to uneven feature retention, making it difficult to simultaneously maintain the contrast of the infrared image and the texture details of the visible light image. SUMMARY

[0008] In view of the uneven feature retention of the existing GAN-based fusion image method when the generator generates a fusion image, it is difficult to simultaneously maintain the contrast of the infrared image and the texture details of the visible light image. And at the same time, most of them use a discriminator, causing the fusion image to fail to sufficiently retain the key features of the other image. The present application provides an infrared and visible light image fusion method based on an improved StyleGAN2, which mainly comprises: S1: constructing an improved StyleGAN2 generator, including a style encoder and a content encoder, the style encoder being used to output a style vector of an image, the content encoder being used to output a content feature of the image, and the generator being used to generate a fusion image; S2: constructing a discriminator for judging the similarity of the fusion image with the infrared image and the similarity of the fusion image with the visible light image; S3: using a loss function to obtain an optimal improved StyleGAN2 for the fusion of the infrared image and the visible light image.

[0009] A computer device comprising a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the steps of the above method.

[0010] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above method.

[0011] A computer program product comprising a computer program or instructions, which, when executed by a processor, implements the steps of the above method.

[0012] The technical scheme provided by the present application has the beneficial effects that the present application provides an infrared and visible light image fusion method based on improved StyleGAN2, realizes more balanced multi-modal feature fusion by means of a double-discriminator fusion framework and by means of collaborative optimization of the feature discrimination capabilities of two modalities. BRIEF DESCRIPTION OF DRAWINGS

[0013] The present application will be further described below in combination with the drawings and embodiments, wherein: Figure 1 is a flowchart of an infrared and visible light image fusion method based on improved StyleGAN2 in an embodiment of the present application; Figure 2 is a generator structure diagram in an embodiment of the present application; Figure 3 is a style encoder structure diagram in an embodiment of the present application; Figure 4 is a content encoder structure diagram in an embodiment of the present application; Figure 5 is a discriminator structure diagram in an embodiment of the present application; Figure 6 is an experimental result diagram in an embodiment of the present application. DETAILED DESCRIPTION

[0014] In order to have a clearer understanding of the technical features, objectives and effects of the present application, the specific embodiments of the present application will now be described in detail with reference to the drawings.

[0015] StyleGAN is an “image generator”, and it can generate a unique and real-looking picture by giving it a random “seed” (called a latent vector). And it finds that different styles of images exist in different resolutions of images, and fine “style” control of the generated picture can be achieved by modifying the style information at the corresponding resolution. StyleGAN2 changes the single style information into style information +, which can inject different styles into the generated image. This feature can solve the problem of infrared and visible light image fusion: the thermal radiation characteristics of infrared images and the texture details of visible light images can be regarded as two independent “styles”, and by selectively injecting these two style information at different levels of the network, a fused image that retains the key features of the two modalities can be generated in theory.

[0016] Embodiment 1 Please refer to Figure 1 , Figure 1 is a flowchart of an infrared and visible light image fusion method based on improved StyleGAN2 in an embodiment of the present application, which specifically includes: S1: Construct an improved StyleGAN2 generator, including a style encoder and a content encoder, the style encoder is used to output the style vector of the image, the content encoder is used for the content features of the image, and the generator is used to generate a fusion image; S2: Construct a discriminator to judge the similarity of the fusion image and the infrared image and the similarity of the fusion image and the visible light image; the similarity output by the discriminator ranges from 0 to 1, and the higher the similarity, the more similar the fusion image is to the real image.

[0017] S3: Use a loss function to obtain an optimal improved StyleGAN2 for the fusion of infrared images and visible light images.

[0018] The specific content is as follows: (1) Generator The generator structure of the improved StyleGAN2 model is shown in Figure 2 , which can be divided into a feature extraction stage, a feature fusion stage, and an image generation stage.

[0019] Style encoder: structure as shown in Figure 3 , input a single-channel image (infrared or visible light grayscale image) with a size of , pass through 5 convolution layers with a step size of 2, the spatial size is constantly compressed, and the depth of the features is constantly increased, the feature map is changed from to . Then a flattening module is used to convert the feature map into a 32768-dimensional vector, and a fully connected layer is used to linearly transform the vector to obtain a 4096-dimensional vector. Finally, a reshaping module is used to convert the vector into a matrix, and finally the style encoder outputs 8 512-dimensional style vectors, which represent different styles hidden in the image from low to high resolution.

[0020] Content encoder: structure as shown in Figure 4 , input a single-channel image (infrared or visible light grayscale image) with a size of , first pass through a , step size 1 convolution to get a feature map, and then pass through 4 convolution layers with a step size of 2 to generate a content feature with a size of . In the encoding process, the feature maps obtained by the first four convolutions are saved, and in the visible light content encoder, they are denoted as , and in the infrared content encoder, they are denoted as .

[0021] The overall process of the generator generating a fusion image is as follows: in the feature extraction stage, the input resolution is The images are single-channel infrared and visible light grayscale images, which are processed by a content encoder and a style encoder, respectively.

[0022] The style encoder generates eight 512-dimensional style vectors, denoted as follows: and .

[0023] The vector output by the content encoder after an image passes through it is the content feature. The vectors generated by the image passing through multiple convolutional modules in the content encoder are combined to form intermediate features. ,in , .

[0024] In the feature fusion stage, the style intersection point is 4, indicating that the first 4 layers of the fused style vector use the infrared style, and the last 4 layers use the visible light style. . Generate fused content features through an attention fusion module. , Generate fusion intermediate features through a CBAM module ,in .

[0025] During the image generation stage, input fusion content features In the synthesis network, the feature maps first pass through four StyledConvBlock modules. The process is as follows: first, the features are upsampled. For example, in the first StyledConvBlock module, the input size is... After upsampling, the size is Then style injection is performed, from the style vector. The first two layers are extracted, and the style vectors are injected into the convolutional layer through an AdaIN module for convolution. At this point, the feature map size is... Then, through a skip connection operation, the intermediate features are... The feature map is added to the feature map, and the output feature is... After four StyledConvBlock modules, and then after one... The convolution operation of the size transforms the feature map into The Tanh activation function is used to normalize the feature values ​​to a range of -1 to 1, ultimately yielding a resolution of [resolution value missing]. The fused image.

[0026] (2) Discriminator The model contains two identical discriminators, used to determine the similarity between the fused image and the infrared image, and the similarity between the fused image and the visible light image, respectively. The discriminator structures are as follows: Figure 5The flow of the discriminator judging the fusion image is shown as follows: The fusion image generated by the generator first passes through a down-sampling module with 4 convolution kernels and a step of 2 to obtain a feature map , and then passes through a flattening operation to become a 32768-dimensional vector. The vector then passes through a fully connected layer to output a scalar between 0 and 1, which represents the probability that the fusion image is a real image according to the discriminator.

[0027] For the infrared discriminator, the output probability when inputting a real infrared image is denoted as , and the output probability when inputting a fusion image is denoted as .

[0028] For the visible light discriminator, the output probability when inputting a real visible light image is denoted as , and the output probability when inputting a fusion image is denoted as .

[0029] (3) Loss function The improved model loss function of StyleGAN2 is as follows: the generator loss function and the discriminator loss function .

[0030] 1 Generator loss function ) The generator loss function is composed of the following parts: content loss , adversarial loss , style loss , and SSIM loss.

[0031] The calculation formula is as follows:

[0032] wherein is a regularization parameter of each loss, used to balance each loss.

[0033] 1.1 Content loss ) The content loss aims to constrain the generator to extract source image information, so that the fusion image retains more details. At the same time, in order to avoid the problem that the model adds useless information in the image to the fusion image in the training process in order to reduce the content loss (for example, there are large black areas in the visible light image at night), a gradient mask-based content loss is introduced.

[0034] The visible light gradient image is , which is a visible light gradient image obtained by processing the visible light image with a sobel operator.

[0035] The infrared gradient map is , which is the infrared gradient image obtained after the infrared image is processed by the sobel operator.

[0036] The visible light gradient mask is , which is 1 in the place where the visible light gradient is larger, otherwise it is 0. The infrared gradient mask is , which is 1 in the place where the infrared gradient is larger, otherwise it is 0. Wherein is an indicator function.

[0037] The content loss is:

[0038] Wherein represents L1 regularization, represents element-wise multiplication.

[0039] 1.2 Adversarial loss ( ) The adversarial loss aims to balance the adversarial relationship between the generator and the two discriminators, prompting the generator to generate a fused image that can deceive the discriminator.

[0040]

[0041] Wherein represents the visible light discriminator , the probability that the generated fused image is judged as a real visible light image, represents the infrared discriminator , the probability that the generated fused image is judged as an infrared visible light image. The constant 1 represents the expected false data generated by the generator to be judged as true by the discriminator.

[0042] 1.3 Style loss ( ) It is used to balance the infrared style and visible light style in the fused image, prompting the generated fused image to have both the texture information of the visible light image and the contrast features of the infrared image.

[0043]

[0044] Wherein represents L1 regularization, respectively are the feature maps obtained after the fused image, the infrared image and the visible light image are extracted by a pre-trained VGG16 network. The VGG16 network does not participate in the training process of the model, and is only used for feature extraction in the style loss.

[0045] The purpose of not directly using the loss calculated by the fusion image and the real image in the style loss is to hope that by comparing the deep features of the fusion image and the real image through the pre-trained VGG16 network, the generated fusion image pays attention to the deep information instead of the shallow pixel value.

[0046] 1.4 SSIM loss ) The SSIM loss belongs to the structural similarity loss, which mainly calculates the brightness, contrast and structural difference between two images.

[0047]

[0048]

[0049] wherein, represents the mean of , represents the mean of , represents the label difference of , represents the label difference of , represents the covariance, represents the fusion image , represents the visible light image or the infrared image , , is a small positive number added to avoid the case where the denominator is 0. The larger the SSIM loss is, the higher the structural similarity is.

[0050] 2 Discriminator loss function ) The improved model of StyleGAN2 has two discriminators, namely the infrared image discriminator and the visible light image discriminator. The discriminator loss function includes the infrared discriminator loss function and the visible light discriminator loss function , and the calculation formula is:

[0051]

[0052]

[0053] wherein represents the probability that the generated fusion image is discriminated as a real visible light image by the visible light discriminator , represents the probability that the generated fusion image is discriminated as a real infrared image by the infrared discriminator The probability of discriminating the real visible light picture as a real visible light image. An infrared discriminator The probability of discriminating the generated fusion picture as an infrared visible light image, An infrared discriminator The probability of discriminating the real infrared picture as an infrared visible light image.

[0054] Experimental results: the model of improved StyleGAN2 is trained using the public data set LLVIP, the training environment is Windows 11 system, the Python version is 3.8, the optimizer is Adam, the generator learning rate is set to 2e-4, the discriminator learning rate is set to 3e-5, and the training is 300 rounds. The test results are as shown in Figure 6 The visible light image, the visible light gray image, the infrared image and the fusion image are compared one by one, so it can be known that the fusion image obtained by the application has the best effect.

[0055] Embodiment 2 A computer device comprising a memory, a processor and a computer program stored on the memory, the processor executing the computer program to implement the steps of the above method.

[0056] Embodiment 3 A computer readable storage medium storing a computer program, when the program is executed by a processor, the steps of the above method are implemented.

[0057] Embodiment 4 A computer program product comprising a computer program or instructions, when the program or instructions are executed by a processor, the steps of the above method are implemented.

[0058] The above only describes the preferred embodiments of the application and is not used to limit the application, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A method for fusing infrared and visible light images based on an improved StyleGAN2, characterized in that, include: S1: Construct an improved generator for StyleGAN2, which includes a style encoder and a content encoder. The style encoder is used to output the style vector of the image, the content encoder is used to generate the content features of the image, and the generator is used to generate the fused image. S2: Construct a discriminator to determine the similarity between the fused image and the infrared image, as well as the similarity between the fused image and the visible light image; S3: Using the loss function, the optimal improved StyleGAN2 is obtained for the fusion of infrared and visible light images.

2. The infrared and visible light image fusion method based on the improved StyleGAN2 as described in claim 1, characterized in that, In S1, the generation process includes a feature extraction stage, a feature fusion stage, and an image generation stage; The process of generating the fused image by the generator is as follows: In the feature extraction stage, a single-channel infrared image and a visible light grayscale image are input. The images are processed by a content encoder and a style encoder, respectively. Multiple style vectors are generated in the style encoder, and content features and intermediate features are generated in the content encoder. In the feature fusion stage, the style intersection point is 4, indicating that the first 4 layers of the fused style vector use infrared style, and the last 4 layers use visible light style, for content features. Generate fused content features through an attention fusion module. intermediate features Generate fusion intermediate features through a CBAM module ,in ; During the image generation stage, input fusion content features In the synthesis network, the feature maps first pass through four StyledConvBlock modules. The process is as follows: first, the features are upsampled, then style is injected from the style vector. The first two layers are extracted, and the style vectors are injected into the convolutional layer through an AdaIN module for convolution. Then, the intermediate features are processed through skip connections. The feature map is added to the feature map to output the feature. After passing through four StyledConvBlock modules, it is then passed through another... The convolution operation of the size is used to normalize the feature values ​​to between -1 and 1 through the Tanh activation function, and finally the fused image is obtained.

3. The infrared and visible light image fusion method based on the improved StyleGAN2 as described in claim 1, characterized in that, In S2, the discriminator includes a visible light discriminator and an infrared discriminator, which have the same structure; The discriminator judges the fused image as follows: The fused image generated by the generator first passes through multiple convolutional kernels and downsampling modules to obtain a feature map. Then, after a flattening operation, the feature map is transformed into a multi-dimensional vector. The vector then passes through a fully connected layer and outputs a scalar between 0 and 1. This scalar represents the probability that the discriminator considers the fused image to be a real image.

4. The infrared and visible light image fusion method based on the improved StyleGAN2 as described in claim 1, characterized in that, In S3, the loss function includes the generator loss function. and discriminator loss function ; Generator loss function Including content loss Combating losses Style loss The SSIM loss is calculated using the following formula: in, These are regularization parameters for various losses, used to balance the various losses; Content loss The calculation formula is: in, Indicates L1 regularization, This indicates element-wise multiplication. Represents a visible light gradient mask. Indicates an infrared gradient mask. Indicates a fused image. Represents a visible light image. Represents an infrared image; Combating losses The calculation formula is as follows: in, Represents a visible light discriminator The probability of identifying the generated fused image as a real visible light image. Indicator of infrared discriminator The probability of identifying the generated fused image as an infrared-visible image; Style loss The calculation formula is: in, These are feature maps obtained by extracting features from the fused image, infrared image, and visible light image using a pre-trained VGG16 network. SSIM loss The calculation formula is: in, express The mean, express The mean, express The annotation difference express The annotation difference Describing covariance, Represents fused images , Representing visible light images or infrared image , , This indicates a preset positive number to avoid a denominator of 0.

5. The infrared and visible light image fusion method based on the improved StyleGAN2 as described in claim 4, characterized in that, Discriminator loss function Including the infrared discriminator loss function and visible light discriminator loss function ; in, Represents a visible light discriminator The probability of identifying the generated fused image as a real visible light image. Represents a visible light discriminator The probability of classifying a real visible light image as a real visible light image. Indicator of infrared discriminator The probability of identifying the generated fused image as an infrared-visible image. Indicator of infrared discriminator The probability of identifying a real infrared image as an infrared visible light image.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes a computer program to implement the steps of the infrared and visible light image fusion method based on the improved StyleGAN2 as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the infrared and visible light image fusion method based on the improved StyleGAN2 as described in any one of claims 1-5.

8. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the steps of the infrared and visible light image fusion method based on the improved StyleGAN2 as described in any one of claims 1-5.

Images