A polarized image denoising method based on self-supervised learning

Abstract

The application discloses a polarization image denoising method based on self-supervised learning, relates to polarization imaging technology, and comprises self-supervised polarization image acquisition, angle-by-angle calculation of Stokes vectors, angle-by-angle calculation of generated noise images, denoising network design and loss function design.The angle-by-angle calculation of Stokes vectors can obtain linear Stokes vectors consistent with the number of polarization angles, the angle-by-angle calculation of generated noise images generates a group of independent polarization images through Stokes vectors according to the polarization relationship, the denoising network design uses multiple sub-networks to process images of each angle, and the loss function design can constrain the network and avoid over-smoothing.The method generates a training image pair by using a single noise polarization image through the polarization relationship, the network can effectively remove noise in the polarization image after self-supervised training, can reach the same level as supervised learning denoising, reduces the dependence of deep learning denoising on a data set, and is favorable to expanding the application scenarios and range of the polarization image denoising technology.

Description

Technical Field

[0001] This invention relates to the field of polarization imaging technology, and in particular to a polarization image denoising method based on self-supervised learning. Background Technology

[0002] Polarization imaging technology can measure and image the polarization state of light, providing rich scene information. Therefore, it has been widely used in remote sensing, biomedical imaging, and materials science. However, polarization images are often affected by noise, which degrades their quality and hinders subsequent image processing algorithms from effectively extracting information. Furthermore, some important polarization parameters, such as the degree of linear polarization (DoLP) and the angle of polarization (AoP), are easily affected by noise, limiting the further application and development of polarization imaging.

[0003] In recent years, supervised learning-based methods have emerged as a promising solution for polarization image denoising, typically employing convolutional neural networks (CNNs) to learn complex denoising models from large-scale datasets. However, supervised learning-based denoising networks are highly dependent on a large number of noisy-sharp image pairs. In practical applications, collecting a large number of noisy-sharp image pairs is time-consuming and labor-intensive, resulting in high acquisition costs, as it requires multiple shots of the same static scene. This also limits the application of supervised learning-based polarization image denoising methods in dynamic scenarios. Furthermore, denoising models learned solely from synthetic images exhibit poor generalization ability in real-world applications. Therefore, while supervised learning-based polarization image denoising methods offer significant denoising effects, they have certain limitations in practical applications due to the difficulty in acquiring suitable datasets. A self-supervised polarization image denoising method is needed that can effectively remove noise from polarization images without requiring extensive dataset creation, thus possessing practical value. Summary of the Invention

[0004] This invention aims to address the problem that supervised learning-based polarization image denoising methods are difficult to apply in practice due to the high cost of dataset acquisition, and provides a self-supervised learning-based polarization image denoising method.

[0005] This invention first introduces a method for creating a self-supervised learning dataset, and then discloses a specific implementation method for polarization image denoising based on self-supervised learning, including:

[0006] 1) Self-supervised polarization image acquisition: For a low signal-to-noise ratio scene, a polarization camera is used to acquire a set of noisy polarization images I1. Then, N different low signal-to-noise ratio scenes are acquired separately to obtain a self-supervised polarization image set {I1, I2, ..., I...} containing only N sets of original noisy images. N};

[0007] 2) Calculate the Stokes vector by angle. For a set of self-supervised polarization images acquired in step 1 containing n (n>=4) angles θ1, θ2, ..., θ n polarization image Starting from the first angle θ1, calculate a set of Stokes vectors for the polarization images at all other angles except θ1. Then, by analogy, the corresponding Stokes vectors are calculated for other angles, resulting in a total of n Stokes vectors.

[0008] In this step, after removing the angle θ to be calculated... j After j∈[1,n], select the three angles with the largest difference to calculate the Stokes vector. This will result in a smaller error in the calculated Stokes vector. It should be noted that there will be multiple solutions when selecting the three angles with the largest difference. In this case, all solutions should be traversed and the average of the calculated Stokes vectors should be taken as the final output Stokes vector.

[0009] 3) Generate noisy images by calculating angles. Based on the original noisy image I in step 2 i The angles, starting from the first angle θ1, use the corresponding Stokes vector. The generated noise image at the first angle is calculated based on the polarization Stokes relation f(·). Then, the Stokes vector of each angle is used to calculate the generated noise image for that angle. Finally, the generated noise images for all angles are merged together to form the generated noise image.

[0010] In this step, the polarization Stokes relation is linear, and the calculation is performed using the three components contained within the Stokes vector. To perform the calculation, the calculation method is as follows: The generated noise image is a polarized image, containing the same polarization angle as the original noise image, and for each angle, the generated noise image is independent of the original noise image.

[0011] 4) Denoising network design: First, select a convolutional neural network f suitable for image denoising tasks. θ (·), then the network is copied n times and arranged in parallel, that is, each network corresponds to a polarization angle. The output of the network is the merging of the four sub-networks, represented as:

[0012] In this step, the denoising network is designed for polarization images. The denoising network contains n sub-networks, each of which requires a polarization image at one angle as input. The number of input channels is 1, and the output channels of all sub-networks at all angles are 1. After output, a merging operation is performed in the channel dimension to obtain a denoised polarization image with n output channels.

[0013] 5) Loss function design: Input the generated noisy image into the denoising network, then calculate the pixel loss between the output and the original noisy image, then input the original noisy image into the denoising network, and calculate the regularization loss between the output and the generated noisy image. This can avoid the denoised image from being too smooth. Finally, combine the two loss functions as the total loss function.

[0014] In this step, the output of the denoising network is first... A pixel loss function is calculated using the mean squared error of the original noisy image I. The original noisy image I is then input into the denoising network, and the network outputs f. c (I) After transformation in steps 2 and 3, and the generated noisy image Calculate a regularized loss function, and the weighted sum of the two loss functions is the final loss function.

[0015] 6) Denoising network training and testing: Feed the generated noisy image into the denoising network designed in step 4, and train it according to the loss function designed in step 5. After training, input the generated noisy image that was not used for denoising network training into the network for testing to obtain the denoised polarization image.

[0016] The beneficial effects and advantages of this invention are as follows:

[0017] 1. This invention discloses a method for creating a self-supervised learning dataset using polarization information. It mainly uses polarization images of other angles that do not contain the generated angle to calculate the polarization image of the angle to be generated through Stokes vectors. This ensures the independence between the generated polarization image and the original polarization image, which can be applied to the training of self-supervised networks. Therefore, when collecting the dataset, only noisy image data needs to be collected, instead of noise-free images, which greatly reduces the acquisition cost and is conducive to the further development of deep learning polarization image denoising technology.

[0018] 2. This invention discloses a self-supervised network design method, which replicates the denoising network according to the number of polarization angles and then arranges them in parallel, with each sub-network corresponding to a polarization angle. The advantage of this is that it can prevent the network from directly learning the identity mapping from the input image to the ground truth image, thus avoiding invalid training.

[0019] 3. This invention discloses a self-supervised loss function design method. First, the paired polarized image is input into the network, and the loss function is calculated by comparing the network output with the noisy polarized image. Then, the noisy polarized image is input into the network, and the loss function is calculated by comparing the network output with the paired polarized image. The advantage of this loss function design is that it can both provide basic constraints for the training of the network and enable effective training, and prevent the image from becoming overly smooth after denoising. Attached Figure Description

[0020] The above and / or additional aspects and advantages of the present invention will become apparent from the following description of the embodiments taken in conjunction with the accompanying drawings.

[0021] Obviously and easy to understand, among which:

[0022] Figure 1 This is a flowchart of the self-supervised learning-based polarization image denoising method of the present invention;

[0023] Figure 2 This is a flowchart of a polarization image denoising method based on self-supervised learning according to an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram illustrating the denoising and polarization information restoration effect of a polarization image according to an embodiment of the present invention. Detailed Implementation

[0025] The embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The present invention is applicable to different color polarization imaging systems (amplitude division, focal plane division, etc.) and different types of polarization information imaging methods (Stokes, Mueller, etc.). The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] The following describes a polarization image denoising method based on self-supervised learning according to an embodiment of the present invention, with reference to the accompanying drawings, taking the Stokes vector imaging method based on focal plane polarization images as an example.

[0027] See appendix Figure 1 and attached Figure 2 The image denoising method in this embodiment includes:

[0028] S1, acquired from a supervised polarization image dataset, is obtained by capturing a scene once using a focal plane polarization camera, resulting in the original noisy image I = [I 0° ,I 45° ,I 90° ,I 135°Furthermore, other scenes are also captured once to obtain the original noise images of different scenes, which are then combined into a self-supervised polarization image dataset. In this embodiment, for ease of understanding and to prevent confusion, only one set of polarization images I is used as an example to explain the present invention.

[0029] S2, Calculate the Stokes vector by angle. In this embodiment, the acquired focal plane polarization image includes polarization images at four angles: 0°, 45°, 90°, and 135°. These four images are represented as I... 0° ,I 45° ,I 90° ,I 135° Then select one of the angles θ j Calculate the Stokes vector for that angle using images from the other three angles. This process can be represented as... Since the calculation process f(·) is a linear operation, the calculation can be simplified to matrix form, i.e. in The parameter matrix is ​​related to the angle, with θ j =135° For example, S 135° The calculation method is as follows:

[0030]

[0031] The last column of the parameter matrix is ​​all 0, which indicates that only the images from the first three angles are available, i.e., I. 0° ,I 45° ,I 90° S participated in the calculation, therefore S 135° with I 135° They are independent of each other.

[0032] S3, generate a noisy image by angle, first select an angle θ j Using the Stokes vector at the corresponding angle The generated noise image at this angle is calculated based on the polarization relationship. The formula expands to:

[0033]

[0034] Then θ j By taking four angles—0°, 45°, 90°, and 135°—and merging them together, a noisy image can be generated. Since the process of generating a noisy image involves linear operations, the steps can be simplified into matrix form, as follows:

[0035]

[0036] As can be seen, the diagonal of the transformation matrix in formula (4) is 0, which means that for each angle, the generated noise image and the original noise image are independent of each other.

[0037] S4, Polarization Image Denoising Network Design: In this embodiment, a residual dense network structure is selected as the basic denoising network. Then, the network is copied four times to obtain four sub-networks, each corresponding to one of the four polarization angles. The output of the network is the combination of four subnetworks, therefore the overall network can be represented as:

[0038]

[0039] Where θ represents the network parameters, and c represents the merged network.

[0040] S5, Loss Function Design: First, generate a noisy image. Input to a denoising network, network output obtains Then, the mean squared error is used to calculate the pixel loss function of the network output and the original noisy image I:

[0041]

[0042] Where ||·||2 represents the mean square error.

[0043] Then the original noisy image is input into the denoising network, and the network outputs f. c (I) The network output image is used to generate the reverse image g(f) according to the calculation process of S2 and S3. c (I) Then the regularization loss function is calculated. Specifically, in one embodiment of the present invention, the regularization loss L... reg :

[0044]

[0045] Where γ is the coefficient of the loss function, with a value of 0.2. Therefore, the final loss function is the sum of the pixel loss and the regularization loss, expressed as:

[0046]

[0047] S6, Denoising Network Training and Testing: Noisy Images Will Be Generated During Training. Input the denoising network and then use the loss function of S5 for constraint. After training, input the noisy polarization image used for testing into the denoising network to obtain the denoised polarization image. Then, reconstruct the light intensity image S0, linear polarization degree map (DoLP), and polarization angle map (AoP) based on the recovered polarization information. Linear polarization degree (DoLP).

[0048] In embodiments of the present invention, a polarization image denoising method based on self-supervised learning is used to remove noise from polarization images. Furthermore, a 3D block matching denoising method (BM3D), a supervised learning-based denoising method (Pol2GT), and two self-supervised denoising methods, Noise2Void (N2V) and Neighbor2Neighbor (N2N), are compared. Figure 3 As shown.

[0049] from Figure 3 As can be seen from the above, the polarization image denoising algorithm based on self-supervised learning proposed in this invention can effectively remove noise from polarization images. Furthermore, it can effectively recover polarization information for polarization degree images and polarization angle images, which are extremely sensitive to noise. Especially for polarization angle images, the method of this invention recovers clearer details and is closer to the true image. Therefore, this invention has good polarization image denoising and polarization information recovery effects. It is also worth noting that the denoising effect of the proposed method is almost identical to that of supervised learning denoising, confirming the beneficialness of this invention. It can reduce the dependence of convolutional neural networks on paired datasets, which is conducive to the further development and application of convolutional neural network denoising technology.

[0050] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0051] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A polarization image denoising method based on self-supervised learning, characterized in that: This method trains a denoising network using unpaired noisy polarization images and includes the following steps: Step 1. Self-supervised polarization image acquisition: For a low signal-to-noise ratio scene, acquire a set of noisy polarization images using a polarization camera. Then, by acquiring N different low signal-to-noise ratio scenes, a self-supervised polarization image set containing only N sets of original noisy images can be obtained. ; Step 2. Calculate the Stokes vector by angle. For a set of n angles in the self-supervised polarization image set acquired in Step 1... polarization image From the first perspective Begin, to remove Calculate a set of Stokes vectors from polarization images at other angles. Then, by analogy, the corresponding Stokes vectors are calculated for other angles, resulting in a total of n Stokes vectors. n>=4; Step 3. Generate a noisy image by calculating angle. Based on the original noisy image in step 2 Angle, from the first angle Begin by using the corresponding Stokes vector. According to the polarization Stokes relation Calculate the generated noise image at the first angle Then, the Stokes vector of each angle is used to calculate the generated noise image for each angle, and finally the generated noise images for all angles are merged together to form the generated noise image. ; Step 4. Denoising Network Design: First, select a convolutional neural network suitable for image denoising tasks. Then, the network is copied n times according to the number of polarization angles. All copies of the network are arranged in parallel, that is, each network corresponds to one polarization angle. The total output of the network is the sum of the outputs of all the networks at all polarization angles, expressed as: ; Step 5. Loss function design: Input the generated noisy image into the denoising network, then calculate the pixel loss by comparing the output of the denoising network with the original noisy image, then input the original noisy image into the denoising network, and calculate the regularization loss by comparing the output of the new denoising network with the generated noisy image to avoid the denoised image being too smooth. Finally, add the two loss functions together with weights to obtain the total loss function. Step 6. Denoising network training and testing: Feed the generated noisy image into the denoising network designed in Step 4, and train it according to the loss function designed in Step 5. After training, input the generated noisy image that was not used for training the denoising network into the network for testing to obtain the denoised polarization image. In calculating the Stokes vector by angle, after removing the angle to be calculated... Then, select the three angles with the largest differences to calculate the Stokes vector. The error of the calculated Stokes vector is smaller than that of the Stokes vector calculated by selecting other angles. It should be noted that there will be multiple solutions when selecting the three angles with the largest differences. In this case, all solutions should be traversed and the average of the calculated Stokes vectors should be taken as the final output Stokes vector.

2. The polarization image denoising method based on self-supervised learning according to claim 1, characterized in that: In the noisy image generated by angle-based calculation, the polarization Stokes relation is linear, and the calculation is performed using the three components contained in the Stokes vector. To perform the calculation, the calculation method is as follows: The generated noise image is a polarized image, containing the same polarization angle as the original noise image, and for each angle, the generated noise image is independent of the original noise image.

3. The polarization image denoising method based on self-supervised learning according to claim 1, characterized in that: In the design of the denoising network, the denoising network is designed for polarized images. The denoising network contains n sub-networks, each of which requires a polarized image at one angle as input. The number of input channels is 1, and the output channels of all sub-networks at all angles are 1. After output, a merging operation is performed in the channel dimension to obtain a denoised polarized image with n output channels.

4. The polarization image denoising method based on self-supervised learning as described in claim 1, characterized in that: In the design of the loss function, the output of the denoising network is first... Compared with the original noisy image A pixel loss function is calculated using mean squared error, and then the original noisy image is... Input the denoising network, then output the network... After transformations in steps 2 and 3, the generated noisy image is compared with the image in steps 3. Calculate a regularized loss function, and the weighted sum of the two loss functions is the final loss function.

Images