Polarization image super-resolution method based on conditional diffusion model

Abstract

The invention discloses a polarization image super-resolution method based on a conditional diffusion model, and the method comprises the following steps: 1, building an active imaging system, and shooting a clear high-resolution polarization image of a target object; step 2, degrading the high-resolution image of the target object into a low-resolution image by using a bicubic down-sampling interpolation method; according to the idea of generating an image by a diffusion model, a conditional diffusion network model is constructed by combining a low-frequency information extraction module, a high-frequency information extraction module, a residual error convolution module and a self-attention mechanism module, a data set is established, and polarization information of the image is fully utilized; the network is trained so that the network can output a high-resolution image according to an input low-resolution polarization image and a noise image obeying standard normal distribution, a good enhancement effect can be achieved by using the conditional diffusion model to carry out super-division on the low-resolution image, and stable performance can be kept in various environments.

Description

Technical Field

[0001] The present invention relates to the fields of deep learning technology and polarization imaging technology, and in particular to a polarization image super-resolution method based on a conditional diffusion model. Background Art

[0002] In modern automotive technology, image super-resolution technology is an important technology in the field of computer vision and image processing, and its research significance is far-reaching and extensive. From the perspective of practical application, image super-resolution can significantly improve the spatial resolution of images, making originally blurred or low-resolution images clear and detailed, which is extremely valuable in many fields such as medical image analysis, security monitoring, remote sensing imaging, and consumer electronics.

[0003] At present, image super-resolution methods are mainly divided into two categories: interpolation-based methods and learning-based methods. Interpolation-based super-resolution methods assume that in a sufficiently small area, the values ​​of adjacent pixels satisfy the smoothness characteristics and have a relatively high similarity with each other. This method increases the resolution of the image by using the information of known pixels and estimating the values ​​of unknown pixels through a certain interpolation function. Learning-based methods use machine learning or deep learning algorithms to predict and restore the high-frequency detail information lost in low-resolution images by learning the mapping relationship between low-resolution images and high-resolution images, thereby generating super-resolution images. As a new deep learning model, the diffusion model has also achieved certain research results in the field of image super-resolution. Both technologies improve the image to a certain extent and improve the image quality. Polarization imaging technology is currently a widely studied image restoration technology, but the enhancement effect is uneven in some application scenarios and environments, and the image quality still does not meet the requirements, which affects further application. Summary of the invention

[0004] In order to overcome the deficiencies in the above-mentioned prior art, the present invention provides a polarization image super-resolution method based on a conditional diffusion model, establishes a data set, makes full use of the polarization information of the image, and trains the network so that it can output a corresponding high-resolution image based on the input low-resolution polarization image and a Gaussian noise image that obeys a standard normal distribution. Using the network to super-resolution the low-resolution polarization image can obtain high-quality high-resolution images, and can maintain good stability in many scenarios.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows: a polarization image super-resolution method based on a conditional diffusion model, comprising the following steps:

[0006] Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object;

[0007] Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image of the target object into a low-resolution image, create a data set, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0008] Step 3: Based on the idea of ​​generating images with a diffusion model, a conditional diffusion neural network is constructed. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain a DOLP image. The low-resolution image and the DOLP image are respectively subjected to a low-frequency feature extraction module and a high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map vector and the high-resolution polarization image with noise are jointly input into the network. First, four encoders are passed through, and the encoder includes two residual convolution modules, two self-attention mechanism modules and a downsampling module to extract features. Then, four decoders are used for upsampling. The decoder includes two residual convolution modules, two self-attention mechanism modules and an upsampling module. Each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the predicted noise is compared with the standard Gaussian noise for mean square error loss to optimize the network.

[0009] Step 4: training the network, using the data set obtained in step 2 to train the conditional diffusion model constructed in step 3 so that it can output a corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution;

[0010] Step 5: Super-resolution the low-resolution polarization image.

[0011] Furthermore, a clear high-resolution polarization image of the target object captured in step 1 is obtained by: the light beam emitted by the light source (1) passes through the polarizer (2) and the beam expander (3) of the polarization modulation system in sequence and then irradiates the target object (4), and after being reflected by the target object (4), reaches the focal plane polarization camera (5), thereby obtaining a high-resolution polarization image of the target object (4).

[0012] Furthermore, in step 1, a clear high-resolution image of the target object is captured, and the high-resolution image is downsampled and degraded using a bicubic interpolation method in digital image processing to obtain a low-resolution polarization image of the target object (6).

[0013] Furthermore, a 532nm blue-green laser is used as the light source (1).

[0014] Furthermore, the residual convolution module in the conditional diffusion neural network described in step three consists of three parts. The first part is a structure consisting of a small batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer; the second part is a module for time embedding, which consists of a Swish activation function and a 1*1 convolution layer; the third part consists of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer.

[0015] Furthermore, the self-attention mechanism module in the conditional diffusion neural network described in step three consists of two parts. The first part is group normalization, which standardizes the input through group normalization to improve the stability and efficiency of training and help reduce internal covariate transfer in the network; the second part is a convolutional projection layer, which is used to generate query (Q), key (K) and value (V) vectors. The query (Q) determines the focus of the current element in the entire sequence; the key (K) provides an identifier for each element to help the query find related elements; the value (V) carries the actual information content, and finally generates an output weighted by the degree of matching between the query and the key.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] According to the idea of ​​generating images by diffusion model, a conditional diffusion network model is constructed by combining low-frequency information extraction module and high-frequency information extraction module, as well as residual convolution module and self-attention mechanism module. A data set is established to make full use of the polarization information of the image. The network is trained so that it can output high-resolution images based on the input low-resolution polarization image and the noise image that obeys the standard normal distribution. Using this conditional diffusion model to super-resolution low-resolution images can achieve good enhancement effects and maintain stable performance in a variety of environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the overall process of a polarization image super-resolution method based on a conditional diffusion model of the present invention;

[0019] Figure 2 is a schematic diagram of an experimental device of the present invention;

[0020] Figure 3 It is a schematic diagram of the residual convolution module structure;

[0021] Figure 4 It is a schematic diagram of the self-attention mechanism module structure;

[0022] Figure 5 : These are the results of the conditional diffusion model super-resolution experiment, (a) is a low-resolution polarization image, (b) is the image enhanced by the present invention, and (c) is a high-resolution image;

[0023] Figure 6 This is a table of super-resolution effect evaluation indicators SSIM and PSNR.

[0024] In the figure: light source 1, polarizer 2, beam expander 3, target object 4, and focal plane polarization camera 5. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] like Figure 1-Figure 6 As shown, the technical solution adopted by the present invention is as follows: This embodiment provides a polarization image super-resolution method based on a conditional diffusion model, comprising the following steps:

[0027] Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object;

[0028] Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image into a low-resolution image, create a data set, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0029] Step 3: Based on the idea of ​​generating images based on the diffusion model, a conditional diffusion neural network is constructed. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain the DOLP image. The low-resolution image and the DOLP image are respectively subjected to the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map and the high-resolution polarization image with noise are jointly input into the network. First, it passes through four encoders, each of which consists of two residual convolution modules, two self-attention mechanism modules and a downsampling module. The encoder will reduce the size of the feature map and increase the dimension. Then it passes through four decoders. The structure of the decoder is roughly the same as that of the encoder. The difference is that the downsampling module in the encoder is replaced by an upsampling module. The decoder will increase the size of the feature map and reduce the dimension. All residual convolution modules in the entire network are used to better perceive features of different dimensions. The self-attention module is used for the model to automatically learn the correlation between different positions when processing time series data, so as to capture richer information. In addition, according to the principles of the forward diffusion process and the backward sampling process of the diffusion model, each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the mean square error loss is calculated between the noise and the standard Gaussian noise to optimize the network.

[0030] Step 4: training the network, using the data set obtained in step 2 to train the constructed conditional diffusion model so that it can output the corresponding high-resolution image according to the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution;

[0031] Step 5: Super-resolution the low-resolution polarization image. Compared with the existing technology, the present invention can super-resolution the low-resolution polarization image into a higher-quality high-resolution image.

[0032] Furthermore, a clear high-resolution polarization image of the target object captured in step one is obtained as follows: the light beam emitted by the light source 1 sequentially passes through the polarizer 2 and the beam expander 3 of the polarization modulation system and then irradiates the target object 4, and after being reflected by the target object 4, reaches the focal plane polarization camera 5, thereby obtaining a high-resolution polarization image of the target object.

[0033] Furthermore, in step 1, a high-resolution image of the target object is captured and obtained, and a bicubic interpolation method in digital image processing is used to downsample and degrade the high-resolution image to obtain a low-resolution polarization image of the target object.

[0034] Furthermore, a 532nm blue-green laser is used as the light source 1 .

[0035] Furthermore, the residual convolution module in the conditional diffusion neural network described in step three is composed of three parts. The first part is a structure composed of a small batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer. This structure can effectively improve the training stability of the model and avoid gradient explosion during training. The second part is a module for time embedding, which is composed of a Swish activation function and a 1*1 convolution layer. It is used to prompt the network the current degree of noise and time step, and improve the network's perception of time steps during sampling. The third part is composed of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer. This part can selectively discard the activation of a part of the neurons in the network, forcing the model to learn more robust features and reduce the overfitting of the model.

[0036] Furthermore, the self-attention mechanism module in the conditional diffusion neural network described in step 3 consists of two parts. The first part is group normalization, which standardizes the input to improve the stability and efficiency of training and help reduce the internal covariate transfer problem in the network. The second part is the convolutional projection layer, which is used to generate query (Q), key (K) and value (V) vectors. The query determines the focus of the current element in the entire sequence; the key provides the identifier of each element to help the query find related elements; the value carries the actual information content, and finally generates the output weighted according to the matching degree of the query and the key.

[0037] Furthermore, the conditional diffusion model described in step three is a model proposed based on the principle of generating images using a diffusion model, and describes the changing relationship between low-resolution images and high-resolution images.

[0038] Specifically, step 1, using an active imaging system, using linear polarized light for active illumination, and capturing a clear high-resolution polarization image of the target 4. In this embodiment, a 532nm blue-green laser is used as the light source 1;

[0039] The light beam emitted by the light source 1 passes through the polarizer 2 and the beam expander 3 of the polarization modulation system in sequence and then irradiates the target 4. After being reflected by the target 4, it reaches the focal plane polarization camera 5. By replacing ten groups of targets 4 with different shapes and patterns, clear high-resolution polarization images are obtained.

[0040] Step 2, establish a data set: use the bicubic downsampling interpolation method to degrade the image obtained in step 1 into a low-resolution image, create a data set, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1;

[0041] Step 3, construct a conditional diffusion neural network. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain the DOLP image. The low-resolution image and the DOLP image are respectively subjected to the low-frequency feature extraction module and the high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map and the high-resolution polarization image with noise are jointly input into the network. First, it passes through four encoders, each of which consists of two residual convolution modules, two self-attention mechanism modules and a downsampling module. The encoder will reduce the size of the feature map and increase the dimension. Then it passes through four decoders. The structure of the decoder is roughly the same as that of the encoder. The difference is that the downsampling module in the encoder is replaced by an upsampling module. The decoder will increase the size of the feature map and reduce the dimension. All residual convolution modules in the entire network are used to better perceive features of different dimensions. The self-attention module is used for the model to automatically learn the correlation between different positions when processing time series data, so as to capture richer information. In addition, according to the principles of the forward diffusion process and the backward sampling process of the diffusion model, each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the mean square error loss is calculated between the noise and the standard Gaussian noise to optimize the network.

[0042] Step 4: Train the network. Use the data set obtained in step 2 to train the constructed conditional diffusion model so that it can output the corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution.

[0043] Step 5: Super-resolution the low-resolution polarization image. Compared with the existing technology, the present invention can super-resolution the low-resolution polarization image into a higher-quality high-resolution image.

[0044] It can be seen from the experimental results that the present invention can effectively perform super-resolution enhancement processing on low-resolution polarization images. Combined with the objective evaluation indicators SSIM (structural similarity) and PSNR (peak signal-to-noise ratio), the image super-resolution effect is remarkable.

[0045] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A polarization image super-resolution method based on a conditional diffusion model, characterized in that: The following steps are involved: Step 1: Build an active imaging system to capture clear, high-resolution polarization images of the target object; Step 2: Use the bicubic downsampling interpolation method to degrade the high-resolution image of the target object into a low-resolution image, create a data set, and divide it into a training set, a validation set, and a test set in a ratio of 7:2:1; Step 3: Based on the idea of ​​generating images with a diffusion model, a conditional diffusion neural network is constructed. The overall network structure is designed in a U-shape. A certain degree of random Gaussian noise associated with time is added to the high-resolution polarization image. The low-resolution polarization image is calculated to obtain a DOLP image. The low-resolution image and the DOLP image are respectively subjected to a low-frequency feature extraction module and a high-frequency feature extraction module to obtain two feature maps. The two are spliced ​​on the channel to form a new feature map vector. The feature map vector and the high-resolution polarization image with noise are jointly input into the network. First, four encoders are passed through, and the encoder includes two residual convolution modules, two self-attention mechanism modules and a downsampling module to extract features. Then, four decoders are used for upsampling. The decoder includes two residual convolution modules, two self-attention mechanism modules and an upsampling module. Each encoder and decoder is embedded with a time parameter associated with the degree of noise added to prompt the network to the degree of denoising required. The network outputs a predicted noise during the training phase, and the predicted noise is compared with the standard Gaussian noise for mean square error loss to optimize the network. Step 4: training the network, using the data set obtained in step 2 to train the conditional diffusion model constructed in step 3 so that it can output a corresponding high-resolution image based on the input low-resolution polarization image and the Gaussian noise image that obeys the standard normal distribution; Step 5: Super-resolution the low-resolution polarization image.

2. The polarization image super-resolution method based on the conditional diffusion model according to claim 1, characterized in that: A clear, high-resolution polarization image of the target object captured in step 1: the light beam emitted by the light source (1) passes through the polarizer (2) and the beam expander (3) of the polarization modulation system in sequence and then irradiates the target object (4), and after being reflected by the target object (4), reaches the focal plane polarization camera (5), thereby obtaining a high-resolution polarization image of the target object (4).

3. The polarization image super-resolution method based on the conditional diffusion model according to claim 2, characterized in that: In step 1, a clear high-resolution image of the target object is captured, and the high-resolution image is downsampled and degraded using a bicubic interpolation method in digital image processing to obtain a low-resolution polarization image of the target object (6).

4. The polarization image super-resolution method based on the conditional diffusion model according to claim 2 or 3, characterized in that: A 532 nm blue-green laser is used as the light source (1).

5. The polarization image super-resolution method based on the conditional diffusion model according to claim 4, characterized in that: The residual convolution module in the conditional diffusion neural network described in step 3 consists of three parts. The first part is a structure consisting of a small batch normalization operation for the feature map, a Swish activation function and a 3*3 convolution layer; the second part is a module for time embedding, which consists of a Swish activation function and a 1*1 convolution layer; the third part consists of a feature batch normalization layer, a Swish activation function, a Dropout regularization layer and a 3*3 convolution layer.

6. The polarization image super-resolution method based on conditional diffusion model according to claim 5, characterized in that: The self-attention mechanism module in the conditional diffusion neural network described in step 3 consists of two parts. The first part is group normalization, which standardizes the input to improve the stability and efficiency of training and help reduce internal covariate transfer in the network; the second part is the convolutional projection layer, which is used to generate query (Q), key (K) and value (V) vectors. The query (Q) determines the focus of the current element in the entire sequence; the key (K) provides an identifier for each element to help the query find related elements; The value (V) carries the actual information content and is ultimately weighted to generate the output based on the degree of match between the query and the key.

Images