An Unsupervised Polarization Image Denoising Method Based on Generative Adversarial Networks

Abstract

The present invention discloses an unsupervised polarization image denoising method based on a generative adversarial network. A generative adversarial network suitable for polarization image denoising is trained using an unpaired data set to obtain a polarization image denoising model with strong generalization performance, including the following steps: building a generative adversarial network; the generative adversarial network includes a denoising generator, a noise generator, an intensity discriminator, a degree of linear polarization discriminator, and a polarization angle discriminator. Among them, both the denoising generator and the noise generator are composed of a shallow feature extraction module, a residual dense module, and a conversion module, and the intensity discriminator, the degree of linear polarization discriminator, and the polarization angle discriminator are all composed of 5 convolutional layers; constructing a loss function and training the generative adversarial network to obtain a generative adversarial network polarization image denoising model; inputting a real polarization noise image into the generative adversarial network polarization image denoising model can achieve unsupervised polarization image denoising.

Description

Technical Field

[0001] The present invention relates to the technical field of polarization imaging, and particularly to an unsupervised polarization image denoising method based on a generative adversarial network. Background Art

[0002] Polarization imaging technology is based on the acquisition and processing of polarization information, which can improve the imaging quality and enhance the perception of target characteristics, and has important value in many fields such as national defense, ocean, and remote sensing. However, when acquiring polarization images under low-light conditions, it is often necessary to obtain high-brightness images through gain, but the gain will amplify the noise of the image, and thus the signal-to-noise ratio of the image is reduced. In addition, due to the non-linear operations during the calculation of the degree of polarization and the polarization angle, the image noise will be further amplified, greatly affecting the acquisition of polarization information. Therefore, polarization image denoising is of great significance. In the field of polarization image denoising, deep learning methods have achieved remarkable results, realizing the removal of image noise and the restoration of polarization information in complex environments. However, the current deep learning polarization image denoising methods are all supervised learning methods, and a large number of strictly paired polarization images are required to train the denoising model. For polarization images, it is costly to obtain a paired image dataset of large-scale noise images and ground-truth images. At the same time, the generalization of the model trained by supervised learning is poor, and its performance will degrade when applied to different environments. For example, when a supervised learning model trained with indoor images processes outdoor noise images, the performance will degrade severely. Unsupervised learning methods are trained in an unpaired manner, learning the conversion between polarization noise images and polarization clear images statistically, rather than one-to-one association, thus overcoming the overfitting problem of supervised learning. Therefore, the unsupervised polarization image denoising technology using unpaired images meets the actual application needs of polarization imaging technology and has important significance. Summary of the Invention

[0003] The present invention aims to solve the problem of using an unpaired dataset to train a neural network to implement polarization image denoising, and provides an unsupervised polarization image denoising method based on a generative adversarial network.

[0004] The object of the present invention is achieved by the following technical solutions:

[0005] An unsupervised polarization image denoising method based on a generative adversarial network, which uses an unpaired dataset to train a generative adversarial network suitable for polarization image denoising to obtain a polarization image denoising model with strong generalization performance, including the following steps:

[0006] S1. Build a generative adversarial network; the generative adversarial network includes a denoising generator, a noise generator, an intensity discriminator, a degree of linear polarization discriminator, and a polarization angle discriminator. Among them, both the denoising generator and the noise generator are composed of a shallow feature extraction module, a residual dense module, and a conversion module. The intensity discriminator, the degree of linear polarization discriminator, and the polarization angle discriminator are all composed of 5 convolutional layers; the denoising generator is used to convert a real polarized noise image into a generated polarized clear image, and the noise generator is used to convert the generated polarized clear image into a generated polarized noise image; the intensity discriminator outputs the probability that the generated polarized clear image is a real image, the degree of linear polarization discriminator outputs the probability that the degree of linear polarization map corresponding to the generated polarized clear image is a real image, and the polarization angle discriminator outputs the probability that the polarization angle map corresponding to the generated polarized clear image is a real image;

[0007] S2. Construct a loss function and train the generative adversarial network; the loss function includes two parts: generator loss and discriminator loss. Among them, the generator loss introduces an adaptive weight and a polarization gradient loss on the basis of the original loss of the generative adversarial network; when training the generative adversarial network, use the unpaired polarized image dataset to train the generative adversarial network unsupervised, and at the same time use the constructed loss function to guide the generative adversarial network to obtain a generative adversarial network polarized image denoising model;

[0008] S3. Input the real polarized noise image into the generative adversarial network polarized image denoising model to achieve unsupervised polarization image denoising.

[0009] Furthermore, the structures of the denoising generator and the noise generator are the same. Both the denoising generator and the noise generator are composed of a shallow feature extraction module, six residual dense modules, and a conversion module;

[0010] Each shallow feature extraction module includes two 3×3 convolutional layers; each residual dense module includes six 3×3 convolutional layers and one 1×1 convolutional layer; each conversion module includes five 3×3 convolutional layers;

[0011] The shallow feature extraction module and the residual dense module are used to extract features from the real polarized noise image input to the denoising generator or the generated polarized clear image input to the noise generator and generate feature maps. The conversion module is used to convert the feature maps into generated polarized clear images or generated polarized noise images;

[0012] The structures of the intensity discriminator, the degree of linear polarization discriminator, and the polarization angle discriminator are the same, and are all composed of 5 4×4 convolutional layers, respectively judging whether the generated polarized clear image and its corresponding degree of linear polarization map and polarization angle map are real images;

[0013] Batch normalization layers and rectified linear unit functions are provided after all convolutional layers of the multi-adversarial network.

[0014] Furthermore, the generator loss constructed in step S2 is specifically as follows:

[0015] The most important part of the generator loss of the generative adversarial network is the generative adversarial loss. To balance the losses of the three discriminators simultaneously, adaptive weights are introduced. Therefore, the generative adversarial loss in the generator loss is:

[0016]

[0017]

[0018]

[0019]

[0020] where x i represents the real polarization noise image, G D represents the denoising generator, D I represents the light intensity discriminator, D DoLP represents the degree of linear polarization discriminator, D AoP represents the polarization angle discriminator, DoLP and AoP respectively represent the functions for calculating the degree of linear polarization and the polarization angle of the image, and N represents the number of image groups input in one training. represents the generative adversarial loss corresponding to the light intensity discriminator, represents the generative adversarial loss corresponding to the degree of linear polarization discriminator, represents the generative adversarial loss corresponding to the polarization angle discriminator, L GAN represents the total generative adversarial loss, w sub represents the weight of the generative adversarial loss corresponding to each discriminator, sub represents the loss type corresponding to the weight; Noise represents the first sub-loss;

[0021] To ensure the authenticity of the polarization information, the gradient losses of the degree of linear polarization and the polarization angle are added to the loss function:

[0022]

[0023]

[0024] where represents the gradient of the image in the horizontal direction, represents the gradient of the image in the vertical direction, G N represents the noise generator, L GradD represents the gradient loss of the degree of linear polarization, represents the gradient loss of the polarization angle.

[0025] Further, the inputs of the light intensity discriminator, the polarization angle discriminator, and the linear polarization degree discriminator are all the real polarization clear image and the generated polarization clear image.

[0026] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the unsupervised polarization image denoising method based on the generative adversarial network are implemented.

[0027] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the unsupervised polarization image denoising method based on the generative adversarial network are implemented.

[0028] Compared with the prior art, the beneficial effects brought by the technical solution of the present invention are as follows:

[0029] 1. The method of the present invention realizes unsupervised learning using unpaired data sets, getting rid of the limitation of the need for a large number of paired data sets in ordinary deep learning methods. At the same time, unsupervised learning uses unpaired data sets to statistically learn the conversion between polarization noise images and polarization clear images, rather than one-to-one associations, thereby overcoming the overfitting problem of supervised learning and significantly improving the generalization ability of the denoising model, enabling effective denoising of polarization images in different environments.

[0030] 2. The present invention provides a generative adversarial network structure suitable for polarization image denoising, adding two discriminators respectively for the linear polarization degree map and the polarization angle map, and at the same time adding a polarization gradient loss, which can better guide the generative adversarial network to recover polarization information, improving the denoising effect on polarization noise images, and at the same time being able to effectively remove the noise of the linear polarization degree map and the polarization angle map corresponding to the polarization noise image. Description of the Drawings

[0031] Figure 1 It is a schematic diagram of the overall framework structure of the generative adversarial network according to an embodiment of the present invention;

[0032] Figure 2a and Figure 2b They are respectively schematic diagrams of the structures of the generator and the discriminator in an embodiment of the present invention;

[0033] Figure 3 It is a flowchart of the unsupervised polarization image denoising method based on the generative adversarial network according to an embodiment of the present invention;

[0034] Figure 4a and Figure 4b They are respectively the denoising effects of the indoor image and the outdoor image collected in an embodiment of the present invention after passing through the generative adversarial network polarization image denoising model. Detailed Embodiments

[0035] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] The unsupervised polarization image denoising method based on a generative adversarial network proposed according to an embodiment of the present invention will be described below with reference to the accompanying drawings.

[0037] See Figure 3 , this embodiment provides an unsupervised polarization image denoising method based on a generative adversarial network, which specifically includes the following steps:

[0038] S1. A non-paired polarization image dataset is obtained by controlling the camera gain and exposure, including real polarization clear images and real polarization noise images. Both the real polarization clear images and the real polarization noise images are captured by a polarization camera and contain intensity information of different polarization angles. To obtain the linear polarization degree map and polarization angle map corresponding to the polarization image, this embodiment uses a split focal plane polarization camera to collect the intensity information of four polarization angles: 0°, 45°, 90°, and 135°. For indoor shooting, the polarization clear images are captured under the conditions of high exposure and no gain, and the real polarization noise images are captured under the conditions of low exposure and high gain. For outdoor shooting, under the condition of fixed exposure, the polarization clear images are captured during the day, and the real polarization noise images are captured at night with fixed gain. For the real polarization clear images, 50 images are continuously obtained in the same scene, and their average light intensity map is used as the real polarization clear image for training. Among them, each pair of real polarization clear images and real polarization noise images is non-paired, that is, the scenes and objects of the two polarization images are not the same. The two polarization images used by the generative adversarial network are both four-channel, so the two polarization images collected are pre-split into sub-images of four different polarization angles and combined into four-channel images for use in the training of the generative adversarial network.

[0039] S2. Build a generative adversarial network. The overall framework diagram of the generative adversarial network is as shown in Figure 1 , and the specific structures of the generator and discriminator of the generative adversarial network are as shown in Figure 2a and Figure 2b . The generative adversarial network includes a denoising generator, a noise generator, a light intensity discriminator, a linear polarization degree discriminator, and a polarization angle discriminator. Among them, the denoising generator and the noise generator have the same structure, and the light intensity discriminator, the linear polarization degree discriminator, and the polarization angle discriminator have the same structure;

[0040] As shown in Figure 2aAs shown in the figure, both the denoising generator and the noise generator are composed of a shallow feature extraction module, six residual dense modules, and a conversion module; each shallow feature extraction module contains two 3×3 convolutional layers; each residual dense module has six 3×3 convolutional layers and one 1×1 convolutional layer; and each conversion module has five 3×3 convolutional layers.

[0041] The shallow feature extraction module and the residual dense module are used to extract features from the real polarization noise image input to the denoising generator or the generated polarization clear image input to the noise generator and generate feature maps, and the conversion module is used to convert the feature maps into generated polarization clear images or generated polarization noise images.

[0042] The three discriminators have the same structure. As Figure 2b shown, they are all composed of five 4×4 convolutional layers; in this embodiment, there is a batch normalization layer and a rectified linear unit function after each convolutional layer of the generative adversarial network.

[0043] S3. The overall training process is as Figure 3 shown. First, input the polarization noise image into the denoising generator to obtain the generated polarization clear image. Then, input the real polarization clear image and the generated polarization clear image into the intensity discriminator together to train the intensity discriminator to recognize the generated polarization clear image and the real polarization clear image. At the same time, input the linear polarization degree map and the polarization angle map corresponding to the real polarization clear image and the generated polarization clear image into the linear polarization degree discriminator and the polarization angle discriminator respectively to train the linear polarization degree discriminator and the polarization angle discriminator to recognize the authenticity of the linear polarization degree map and the polarization angle map. While training the three discriminators, train the denoising generator so that the generated polarization clear image output by it is recognized as a real polarization clear image by the three discriminators. At the same time, to ensure the continuity of the denoising structure of the generated polarization clear image and the structure of the real polarization noise image, input the generated polarization clear image into the noise generator and train the noise generator so that its output is close to the real polarization noise image.

[0044] S4. Design the loss function. The discriminator loss in the training process of the generative adversarial network is as follows:

[0045]

[0046]

[0047]

[0048] where x i represents the real polarization noise image, y i represents the real polarization clear image, G D represents the denoising generator, D I represents the intensity discriminator, D DoLPDenotes the linear polarization degree discriminator, D AoP Denotes the polarization angle discriminator, DoLP and AoP denote the linear polarization degree and polarization angle of the calculated image, N denotes the number of images input for one training, Denotes the discriminator loss corresponding to the light intensity discriminator, Denotes the discriminator loss corresponding to the linear polarization degree discriminator, Denotes the discriminator loss corresponding to the polarization angle discriminator;

[0049] For the generator loss, the loss consists of the generative adversarial loss, the cycle loss, and the intrinsic loss, where the generative adversarial loss is:

[0050]

[0051]

[0052]

[0053] [[ID=CO]]

[0054] Where Denotes the generative adversarial loss corresponding to the light intensity discriminator, Denotes the generative adversarial loss corresponding to the linear polarization degree discriminator, Denotes the generative adversarial loss corresponding to the polarization angle discriminator, L GAN Denotes the total generative adversarial loss, w sub Denotes the weight of the generative adversarial loss corresponding to each discriminator, sub represents the loss type corresponding to the weight; Noise represents the first sub-loss.

[0055] The cycle loss is:

[0056]

[0057] Where G N Is the noise generator, L cyc Denotes the cycle loss. In addition, to ensure the authenticity of the polarization information, the gradient losses of the linear polarization degree and the polarization angle are added:

[0058]

[0059]

[0060] Where Denotes the gradient of the calculated image in the horizontal direction, Denotes the gradient of the calculated image in the vertical direction, L[[ID=6CO]] GradD Denotes the gradient loss of the linear polarization degree, Denotes the gradient loss of the polarization angle;

[0061] Finally, an intrinsic loss is added to ensure that the polarization information is not distorted:

[0062]

[0063] where L idt represents the intrinsic loss;

[0064] The final total loss is:

[0065] L total = λ1L GAN + λ2L cyc + λ3L GradD + λ4L GradA + λ5L idt

[0066] where λ1, λ2, λ3, λ4, and λ5 represent the weight coefficients of each loss, and L total represents the total generator loss;.

[0067] In the embodiments of the present invention, during the training process, the real polarization noise images and real polarization clear images will be flipped and rotated to achieve data augmentation. Specifically, in this embodiment, the number of training samples per time is 36, the learning rate is initialized to 0.0001. If the generator loss does not decrease after more than 5 epochs, the learning rate is reduced to one-fifth of the original. The training epoch is 90, and the Adam algorithm is used to optimize the loss function.

[0068] In the embodiments of the present invention, the intensity map, degree of linear polarization map, and polarization angle map output by the polarization image denoising model of the generative adversarial network obtained after training are as shown in Figure 4a and Figure 4b which include the processing effects of both indoor and outdoor images. Among them, Figure A is the noise intensity map, Figure B is the noise degree of linear polarization map, Figure C is the noise polarization angle map, and Figures D, E, and F are the intensity map, noise degree of linear polarization map, and polarization angle map after denoising using the embodiments of the present invention, respectively. It can be seen from Figure 4a and Figure 4b that the unsupervised polarization image denoising method based on the generative adversarial network proposed by the present invention has good denoising effects on both indoor and outdoor images, and can well restore the polarization information for the polarization degree image and polarization angle image that are sensitive to noise.

[0069] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0070] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the spirit of the present invention and the scope protected by the claims, those of ordinary skill in the art can make many specific variations in form under the inspiration of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. An unsupervised polarization image denoising method based on generative adversarial networks, characterized in that Train a generative adversarial network using unpaired datasets to obtain a polarization image denoising model with strong generalization performance, including the following steps: S1. Build a generative adversarial network; the generative adversarial network includes a denoising generator, a noise generator, an intensity discriminator, a degree of linear polarization discriminator, and a polarization angle discriminator. The denoising generator and the noise generator are both composed of a shallow feature extraction module, a residual dense module, and a conversion module. The intensity discriminator, the degree of linear polarization discriminator, and the polarization angle discriminator are all composed of 5 convolutional layers; the denoising generator is used to convert a real polarization noise image into a generated polarization clear image, and the noise generator is used to convert the generated polarization clear image into a generated polarization noise image; the intensity discriminator outputs the probability that the generated polarization clear image is a real image, the degree of linear polarization discriminator outputs the probability that the degree of linear polarization map corresponding to the generated polarization clear image is a real image, and the polarization angle discriminator outputs the probability that the polarization angle map corresponding to the generated polarization clear image is a real image; the inputs of the intensity discriminator, the polarization angle discriminator, and the degree of linear polarization discriminator are all real polarization clear images and generated polarization clear images; S2. Construct a loss function and train the generative adversarial network; the loss function includes two parts: generator loss and discriminator loss. Among them, the generator loss introduces an adaptive weight and a polarization gradient loss on the basis of the original loss of the generative adversarial network; when training the generative adversarial network, use an unpaired polarization image dataset for unsupervised training, and at the same time use the constructed loss function to guide the generative adversarial network to obtain the generative adversarial network; S3. Input the real polarization noise image into the denoising generator of the generative adversarial network to achieve unsupervised polarization image denoising.

2. The unsupervised polarization image denoising method based on a generative adversarial network according to claim 1, characterized in that the denoising generator and the noise generator have the same structure, and the denoising generator and the noise generator are both composed of a shallow feature extraction module, six residual dense modules, and a conversion module; each shallow feature extraction module includes two 3×3 convolutional layers; each residual dense module includes six 3×3 convolutional layers and one 1×1 convolutional layer; each conversion module includes five 3×3 convolutional layers; the shallow feature extraction module and the residual dense module are used to extract features from the real polarization noise image input to the denoising generator or the generated polarization clear image input to the noise generator and generate feature maps, and the conversion module is used to convert the feature maps into generated polarization clear images or generated polarization noise images; the intensity discriminator, the degree of linear polarization discriminator, and the polarization angle discriminator have the same structure, and are all composed of 5 4×4 convolutional layers, respectively discriminating whether the generated polarization clear image and its corresponding degree of linear polarization map and polarization angle map are real images; After all convolutional layers of the multi-resistant network are provided with batch normalization layers and rectified linear unit functions.

3. The unsupervised polarization image denoising method based on a generative adversarial network according to claim 1, wherein The generator loss constructed in step S2 is specifically as follows: The most important part of the generator loss of the generative adversarial network is the generative adversarial loss. To balance the losses of the three discriminators simultaneously, adaptive weights are introduced. Therefore, the generative adversarial loss in the generator loss is as follows: ； ； ； ； Among them represents the true polarization noise image represents the denoising generator represents the light intensity discriminator represents the degree of linear polarization discriminator represents the polarization angle discriminator and respectively represent the functions for calculating the degree of linear polarization and the polarization angle of the image represents the number of image groups for one training input represents the generative adversarial loss corresponding to the light intensity discriminator represents the generative adversarial loss corresponding to the degree of linear polarization discriminator represents the generative adversarial loss corresponding to the polarization angle discriminator represents the total generative adversarial loss represents the weight of the generative adversarial loss corresponding to each discriminator characterizes the loss type corresponding to the weight; Noise represents the first sub-loss To ensure the authenticity of the polarization information, gradient losses of the degree of linear polarization and the polarization angle are added to the loss function: ； ； Among them represents calculating the gradient of the image in the horizontal direction represents calculating the gradient of the image in the vertical direction represents a noise generator represents the gradient loss of the degree of linear polarization represents the gradient loss of the polarization angle 4. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, the steps of the unsupervised polarization image denoising method based on the generative adversarial network according to any one of claims 1 to 3 are implemented.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the unsupervised polarization image denoising method based on the generative adversarial network according to any one of claims 1 to 3 are implemented.

Images