A polarization image super-resolution reconstruction method based on conditional diffusion model

Abstract

The present invention discloses a polarization image super-resolution method based on a conditional diffusion model, comprising the following steps: step 1, building an active imaging system to capture a clear high-resolution polarization image of a target object; step 2, using a bicubic downsampling interpolation method to degenerate the high-resolution image of the target object into a low-resolution image; based on the idea of ​​generating images using a diffusion model, a conditional diffusion network model is constructed by combining a low-frequency information extraction module and a high-frequency information extraction module, as well as a residual convolution module and a self-attention mechanism module, a data set is established, the polarization information of the image is fully utilized, and the network is trained so that it can output a high-resolution image based on an input low-resolution polarization image and a noise image that obeys a standard normal distribution. Using the conditional diffusion model to perform super-resolution on the low-resolution image can achieve good enhancement effects and maintain stable performance under various environments.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning technology and polarization imaging technology, and in particular to a polarization image super-resolution method based on a conditional diffusion model. Background Art

[0002] In modern automotive technology, image super-resolution, a key technology in computer vision and image processing, has profound and extensive research implications. From a practical application perspective, image super-resolution can significantly improve the spatial resolution of an image, rendering previously blurry or low-resolution images clear and detailed. This has significant value in a variety of fields, including medical image analysis, security monitoring, remote sensing imaging, and consumer electronics.

[0003] Currently, image super-resolution methods are mainly divided into two categories: interpolation-based methods and learning-based methods. Interpolation-based super-resolution methods assume that within a sufficiently small area, the values ​​of adjacent pixels satisfy a smooth property and have a high degree of similarity. This method uses information from known pixels and an interpolation function to estimate the values ​​of unknown pixels, thereby increasing the image resolution. Learning-based methods use machine learning or deep learning algorithms to learn the mapping relationship between low-resolution and high-resolution images to predict and restore high-frequency details lost in the low-resolution image, thereby generating a super-resolution image. Diffusion models, as a new deep learning model, have also achieved certain research results in the field of image super-resolution. Both techniques improve the image quality to a certain extent. Polarization imaging technology is a widely studied image restoration technology, but the enhancement effect varies in certain application scenarios and environments, and the image quality still does not meet the requirements, which hinders its further application. Summary of the Invention

[0004] In order to overcome the deficiencies in the above-mentioned prior art, the present invention provides a polarization image super-resolution method based on a conditional diffusion model. A data set is established to fully utilize the polarization information of the image. The network is trained so that it can output a corresponding high-resolution image based on the input low-resolution polarization image and a Gaussian noise image that obeys a standard normal distribution. Using this network to super-resolve low-resolution polarization images can obtain high-quality high-resolution images, and can maintain good stability in many scenarios.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a polarization image super-resolution method based on a conditional diffusion model, comprising the following steps:

[0006] Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object;

[0007] Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image of the target object into a low-resolution image, create a dataset, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0008] Step 3. Based on the idea of ​​generating images based on the diffusion model, a conditional diffusion neural network is constructed. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain a DOLP image. The low-resolution image and the DOLP image are respectively subjected to the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map vector and the noisy high-resolution polarization image are jointly input into the network; first, it passes through four encoders, which include two residual convolution modules, two self-attention mechanism modules and a downsampling module to extract features, and then it is upsampled through four decoders. The decoder includes two residual convolution modules, two self-attention mechanism modules and an upsampling module. Each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the predicted noise is compared with the standard Gaussian noise to perform mean square error loss to optimize the network;

[0009] Step 4: Train the network. Use the data set obtained in step 2 to train the conditional diffusion model constructed in step 3 so that it can output the corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution.

[0010] Step 5: Super-resolution the low-resolution polarization image.

[0011] Furthermore, a clear high-resolution polarization image of the target object captured in step 1 is obtained by: the light beam emitted by the light source (1) passes through the polarizer (2) and the beam expander (3) of the polarization modulation system in sequence and then irradiates the target object (4), and then reaches the focal plane polarization camera (5) after being reflected by the target object (4), thereby obtaining a high-resolution polarization image of the target object (4).

[0012] Furthermore, a clear high-resolution image of the target object is obtained by shooting in step 1, and the high-resolution image is downsampled and degraded using a bicubic interpolation method in digital image processing to obtain a low-resolution polarization image of the target object (4).

[0013] Furthermore, a 532nm blue-green laser is used as the light source (1).

[0014] Furthermore, the residual convolution module in the conditional diffusion neural network described in step 3 consists of three parts. The first part is a structure consisting of a small-batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer; the second part is a module for time embedding, which consists of a Swish activation function and a 1*1 convolution layer; the third part consists of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer.

[0015] Furthermore, the self-attention mechanism module in the conditional diffusion neural network described in step 3 consists of two parts. The first part is group normalization, which standardizes the input through group normalization to improve the stability and efficiency of training and help reduce internal covariate transfer in the network; the second part is the convolutional projection layer, which is used to generate query (Q), key (K) and value (V) vectors. The query (Q) determines the focus of the current element in the entire sequence; the key (K) provides an identifier for each element to help the query find related elements; the value (V) carries the actual information content, and finally generates the output based on the weighted matching degree between the query and the key.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] Based on the idea of ​​generating images by diffusion model, a conditional diffusion network model is constructed by combining low-frequency information extraction module and high-frequency information extraction module, as well as residual convolution module and self-attention mechanism module. A data set is established to make full use of the polarization information of the image. The network is trained so that it can output high-resolution images based on the input low-resolution polarization image and the noise image that obeys the standard normal distribution. Using this conditional diffusion model to super-resolve low-resolution images can achieve good enhancement effects and maintain stable performance in a variety of environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the overall process of a polarization image super-resolution method based on a conditional diffusion model of the present invention;

[0019] Figure 2 is a schematic diagram of the experimental device of the present invention;

[0020] Figure 3 It is a schematic diagram of the residual convolution module structure;

[0021] Figure 4 This is a schematic diagram of the self-attention mechanism module structure;

[0022] Figure 5 The results of the conditional diffusion model super-resolution experiment are shown in Figure 1. (a) is a low-resolution polarization image, (b) is the image enhanced by the present invention, and (c) is a high-resolution image.

[0023] Figure 6 This is a table of super-resolution effect evaluation indicators SSIM and PSNR.

[0024] In the figure: light source 1, polarizer 2, beam expander 3, target 4, and focal plane polarization camera 5. DETAILED DESCRIPTION

[0025] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] like Figures 1-6 As shown, the technical solution adopted by the present invention is as follows: This embodiment provides a polarization image super-resolution method based on a conditional diffusion model, comprising the following steps:

[0027] Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object;

[0028] Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image into a low-resolution image, create a dataset, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0029] Step 3: Based on the idea of ​​generating images based on the diffusion model, a conditional diffusion neural network is constructed. The overall network structure follows a U-shaped design. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The DOLP image is calculated on the low-resolution polarization image. The low-resolution image and the DOLP image are respectively processed through the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. This feature map and the noisy high-resolution polarization image are jointly input into the network. First, it passes through four encoders, each of which consists of two residual convolution modules, two self-attention mechanism modules and a downsampling module. The encoder will reduce the size of the feature map and increase the dimension. Then it passes through four decoders. The structure of the decoder is roughly the same as that of the encoder. The difference is that the downsampling module in the encoder is replaced by an upsampling module. The decoder will increase the size of the feature map and reduce the dimension. All residual convolution modules in the entire network are used to better perceive features of different dimensions. The self-attention module is used for the model to automatically learn the correlation between different positions when processing time series data, so as to capture richer information. In addition, according to the principles of the forward diffusion process and backward sampling process of the diffusion model, each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the mean square error loss is performed on the noise compared with the standard Gaussian noise to optimize the network.

[0030] Step 4: Train the network. Use the data set obtained in step 2 to train the constructed conditional diffusion model so that it can output the corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution.

[0031] Step 5: Super-resolution the low-resolution polarization image. Compared with the existing technology, the present invention can super-resolve the low-resolution polarization image into a higher-quality high-resolution image.

[0032] Furthermore, a clear, high-resolution polarization image of the target object captured in step 1 is obtained: the light beam emitted by the light source 1 passes through the polarizer 2 and the beam expander 3 of the polarization modulation system in sequence and then irradiates the target object 4, and then reaches the focal plane polarization camera 5 after being reflected by the target object 4, thereby obtaining a high-resolution polarization image of the target object.

[0033] Furthermore, in step 1, a high-resolution image of the target object is obtained by shooting, and the high-resolution image is downsampled and degraded using a bicubic interpolation method in digital image processing to obtain a low-resolution polarization image of the target object.

[0034] Furthermore, a 532nm blue-green laser is used as the light source 1 .

[0035] Furthermore, the residual convolution module in the conditional diffusion neural network described in step three consists of three parts. The first part is a structure consisting of a small-batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer. This structure can effectively improve the training stability of the model and avoid gradient explosion during training. The second part is a module for time embedding, which consists of a Swish activation function and a 1*1 convolution layer. It is used to prompt the network the current degree of noise added and the time step, thereby improving the network's perception of the time step during the sampling process. The third part consists of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer. This part can selectively discard the activation of a part of the neurons in the network, forcing the model to learn more robust features and reduce the overfitting of the model.

[0036] Furthermore, the self-attention mechanism module in the conditional diffusion neural network described in step 3 consists of two parts. The first part is group normalization, which standardizes the input to improve training stability and efficiency and help reduce internal covariate shift problems in the network. The second part is a convolutional projection layer, which is used to generate query (Q), key (K), and value (V) vectors. The query determines the focus of the current element in the entire sequence; the key provides an identifier for each element, helping the query find relevant elements; the value carries the actual information content and is ultimately weighted according to the degree of match between the query and the key to generate the output.

[0037] Furthermore, the conditional diffusion model described in step three is a model proposed based on the principle of image generation using a diffusion model, and describes the changing relationship between low-resolution images and high-resolution images.

[0038] Specifically, step 1: using an active imaging system, using linearly polarized light for active illumination, to capture a clear, high-resolution polarization image of the target 4. In this embodiment, a 532nm blue-green laser is used as the light source 1;

[0039] The light beam emitted by light source 1 passes through polarizer 2 and beam expander 3 of the polarization modulation system before being irradiated onto target 4. After being reflected by target 4, it reaches focus plane polarization camera 5. By changing ten groups of targets 4 with different shapes and patterns, clear, high-resolution polarization images are obtained.

[0040] Step 2: Create a dataset: Use the bicubic downsampling interpolation method to degrade the image obtained in step 1 into a low-resolution image to create a dataset, which is divided into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0041] Step 3: Construct a conditional diffusion neural network. The overall network structure follows a U-shaped design. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The DOLP image is calculated from the low-resolution polarization image. The low-resolution image and the DOLP image are respectively processed through the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. This feature map and the noisy high-resolution polarization image are jointly input into the network. First, it passes through four encoders, each of which consists of two residual convolution modules, two self-attention mechanism modules and a downsampling module. The encoder will reduce the size of the feature map and increase the dimension. Then it passes through four decoders. The structure of the decoder is roughly the same as that of the encoder. The difference is that the downsampling module in the encoder is replaced by an upsampling module. The decoder will increase the size of the feature map and reduce the dimension. All residual convolution modules in the entire network are used to better perceive features of different dimensions. The self-attention module is used for the model to automatically learn the correlation between different positions when processing time series data, so as to capture richer information. In addition, according to the principles of the forward diffusion process and backward sampling process of the diffusion model, each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the mean square error loss is performed on the noise compared with the standard Gaussian noise to optimize the network.

[0042] Step 4: Train the network. Use the data set obtained in step 2 to train the constructed conditional diffusion model so that it can output the corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution.

[0043] Step 5: Super-resolution the low-resolution polarization image. Compared with the existing technology, the present invention can super-resolve the low-resolution polarization image into a high-resolution image with higher quality.

[0044] It can be seen from the experimental results that the present invention can effectively perform super-resolution enhancement processing on low-resolution polarization images. Combined with the objective evaluation indicators SSIM (structural similarity) and PSNR (peak signal-to-noise ratio), the image super-resolution effect is significant.

[0045] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A polarization image super-resolution reconstruction method based on a conditional diffusion model, characterized in that: The following steps are involved: Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object; Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image of the target object into a low-resolution image, create a dataset, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1; Step 3. Based on the idea of ​​generating images based on the diffusion model, a conditional diffusion neural network is constructed. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain a DOLP image. The low-resolution image and the DOLP image are respectively subjected to the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map vector and the noisy high-resolution polarization image are jointly input into the network; first, it passes through four encoders, which include two residual convolution modules, two self-attention mechanism modules and a downsampling module to extract features, and then it is upsampled through four decoders. The decoder includes two residual convolution modules, two self-attention mechanism modules and an upsampling module. Each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the predicted noise is compared with the standard Gaussian noise to perform mean square error loss to optimize the network; Step 4: Train the network. Use the data set obtained in step 2 to train the conditional diffusion model constructed in step 3 so that it can output the corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution. Step 5: Perform super-resolution reconstruction on the low-resolution polarization image.

2. The polarization image super-resolution reconstruction method based on the conditional diffusion model according to claim 1, characterized in that: A clear, high-resolution polarization image of the target object captured in step 1: The light beam emitted by the light source (1) passes through the polarizer (2) and the beam expander (3) of the polarization modulation system in sequence and then irradiates the target object (4). After being reflected by the target object (4), the light beam reaches the focal plane polarization camera (5), thereby obtaining a high-resolution polarization image of the target object (4).

3. The polarization image super-resolution reconstruction method based on the conditional diffusion model according to claim 2, characterized in that: In step 1, a clear high-resolution image of the target object is obtained by shooting, and the high-resolution image is downsampled and degraded using the bicubic interpolation method in digital image processing to obtain a low-resolution polarization image of the target object (4).

4. The polarization image super-resolution reconstruction method based on the conditional diffusion model according to claim 2 or 3, characterized in that: A 532 nm blue-green laser is used as the light source (1).

5. The polarization image super-resolution reconstruction method based on the conditional diffusion model according to claim 4, characterized in that: The residual convolution module in the conditional diffusion neural network described in step 3 consists of three parts. The first part is a structure consisting of a small batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer; the second part is a module for time embedding, which consists of a Swish activation function and a 1*1 convolution layer; the third part consists of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer.

6. The polarization image super-resolution reconstruction method based on the conditional diffusion model according to claim 5, characterized in that: The self-attention mechanism module in the conditional diffusion neural network described in step 3 consists of two parts. The first part is group normalization, which standardizes the input to improve the stability and efficiency of training and help reduce internal covariate shift in the network. The second part is the convolutional projection layer, which is used to generate the query (Q), key (K), and value (V) vectors. The query (Q) determines the focus of the current element in the entire sequence; the key (K) provides an identifier for each element, helping the query find related elements. The value (V) carries the actual information content and is ultimately weighted to generate the output based on the degree of match between the query and the key.

Images