A super-resolution reconstruction method for polarization images of split focal planes based on attention mechanism

Abstract

A method for super-resolution reconstruction of polarization images of focal planes based on an attention mechanism. The invention belongs to the technical field of optical image processing, and specifically relates to the technical field of super-resolution reconstruction of polarization images of focal planes. The method comprises the following steps: S1, using a polarization detector of a focal plane to collect images and construct an image data set; S2, constructing a super-resolution neural network model, in which a channel-adaptive sub-pixel convolution module is used to realize sub-pixel reconstruction of feature images of any number of channels, and at the same time, the channel weight is adaptively adjusted according to the proportion of feature information in different polarization directions; S3, using the image data set to train the super-resolution neural network model; S4, verifying the super-resolution neural network model according to evaluation indicators; S5, using the super-resolution neural network model to perform super-resolution reconstruction of polarization images.

Description

Technical Field

[0001] The invention belongs to the technical field of optical image processing, and in particular to the technical field of super-resolution reconstruction of focal plane polarization images. Background Art

[0002] The polarization characteristics of light reflected from an object are closely related to factors such as its surface roughness, geometry, material and texture structure. Even if different objects have the same intensity reflectivity, the polarization characteristics of their reflected light will vary greatly. Ordinary imaging systems recognize targets by distinguishing the difference in light intensity between the target and the environment in the scene. When the target and the surrounding environment have similar light intensity, the target information will be overwhelmed by the complex background signal. As an emerging detection technology, polarization imaging technology can significantly improve the detection capability in low-contrast and low-visibility environments by obtaining the polarization information of the target. It has broad application prospects in military reconnaissance, medical testing, defogging imaging and other fields.

[0003] Unlike light intensity information, existing detectors cannot directly detect and record the polarization information of scene reflected light. Therefore, it is necessary to collect multiple polarization images at the same time and use the Stokes vector to solve polarization characteristics such as polarization degree and polarization angle. At present, there are two relatively mature polarization imaging systems: time-sharing modulation and simultaneous modulation. Simultaneous modulation obtains multiple polarization component images of the target in a time series, requiring the target to be in a static state and the light radiation environment to remain unchanged. Although the time-sharing full polarization imaging technology based on liquid crystal modulation can increase the frame rate of image acquisition, it will still generate false polarization information when tracking and shooting moving targets, and the intensity image cannot be correctly enhanced. The simultaneous polarization imaging technology based on the split focal plane detector can obtain four different polarization component images of the target through one exposure. It is less disturbed by the external environment, has a fast detection speed, and there are no moving devices in the system. The stability and reliability are significantly improved, which is more suitable for real-time tracking and imaging of long-distance moving targets. Compared with the simultaneous polarization imaging system such as split aperture and split amplitude, it reduces the difficulty of designing and assembling the optical-mechanical system, and has obvious advantages in system miniaturization.

[0004] The main limitation of the current focal plane polarization imaging system is that the staggered arrangement of micro-linear polarizers causes the imaging resolution to drop to 1 / 4 of the detector, making it impossible to perform "point-to-point" Stokes calculations, which affects the inversion accuracy of polarization characteristics. Therefore, it is necessary to perform super-resolution reconstruction of the focal plane polarization image. The traditional interpolation method for super-resolution reconstruction does not fully utilize the characteristic information of the polarization image, and the reconstruction result is not ideal; the super-resolution reconstruction based on deep learning only focuses on the light intensity information, usually reconstructing a single image into a high-resolution image, and cannot reconstruct high-resolution polarization images of four different polarization directions at a time. It does not fully utilize the characteristics of the focal plane polarization image, which reduces both the reconstruction efficiency and the reconstruction accuracy of the super-resolution image. Summary of the invention

[0005] In order to solve the technical problem that the previous super-resolution reconstruction method of split-focal plane polarization imaging cannot reconstruct high-resolution polarization images of four different polarization directions at one time, does not fully utilize the characteristics of the split-focal plane polarization image, and reduces both the reconstruction efficiency and the reconstruction accuracy of the super-resolution image, the present invention provides a super-resolution reconstruction method of split-focal plane polarization image based on an attention mechanism.

[0006] The method comprises the following steps:

[0007] S1, using a focal plane polarization detector to collect images and construct an image data set;

[0008] S2. Constructing a super-resolution neural network model, in which a channel-adaptive sub-pixel convolution module is used to realize sub-pixel reconstruction of feature images of any number of channels, and at the same time, the channel weight is adaptively adjusted according to the proportion of feature information in different polarization directions;

[0009] S3, using image datasets to train super-resolution neural network models;

[0010] S4. Verify the super-resolution neural network model according to the evaluation indicators;

[0011] S5. Use the super-resolution neural network model to perform super-resolution reconstruction of polarization images.

[0012] Further, step S1 is specifically as follows:

[0013] S11, using a focal plane polarization detector to collect images, split the images into linear polarization images with four polarization directions of 0°, 45°, 90° and 135°, and regard these four images as high-resolution images as the output of the super-resolution network model;

[0014] S12, performing interval downsampling on the four split images, and recombining the four images after the interval downsampling into a component focal plane polarization image as an input of the super-resolution network model;

[0015] S13. Construct an image data set including output data and corresponding input data according to the methods of step S11 and step S12 respectively.

[0016] Furthermore, the super-resolution neural network model includes an input layer, a convolution layer, a dense residual module, a spatial attention module, a channel adaptive sub-pixel convolution module and an output layer in sequence;

[0017] The focal plane polarization image is taken as input, the convolution layer is used to extract shallow features, the dense residual module is used to extract deep features, the spatial attention module is used to extract the attention map of the feature map and perform weighted fusion, and the channel-adaptive sub-pixel convolution module is used to upsample the extracted feature map to obtain four linear polarization images with different polarization directions.

[0018] Furthermore, the convolution layer is composed of 96 5×5 convolution kernels.

[0019] Furthermore, the dense residual module has a total of 12 residual blocks, each residual block is composed of a 3×3 convolutional layer, a batch normalization layer and a linear rectifier unit ReLU activation function, and the residual connection is realized by a 1×1 convolutional layer; the residual blocks are connected using a 1×1 convolutional layer and an average pooling layer.

[0020] Furthermore, the spatial attention module performs channel maximum pooling and channel average pooling on the feature map to obtain two single-channel feature maps, combines the two single-channel feature maps to obtain a dual-channel feature map, performs convolution operation on the dual-channel feature map using a 5×5 convolution kernel and activates it using a Sigmoid activation function to obtain an attention map composed of weight coefficients, and weightedly fuses the attention map with each feature map to obtain a new feature map.

[0021] Furthermore, the channel adaptive sub-pixel convolution module performs maximum pooling and average pooling on the feature map obtained in the previous layer to obtain channel features of length c, and inputs the obtained channel features into a channel adaptive sub-network to obtain channel weights, wherein the channel adaptive sub-network is sequentially composed of two convolutional layers and a fully connected layer, wherein the number of neurons in the fully connected layer is equal to the number of channels of the feature map, and a weighted feature map is obtained according to the channel weight calculation, and then the weighted feature map is conditionally batch normalized, wherein the four polarization directions of 0°, 45°, 90° and 135° are used as scaling factors and bias terms for conditionally adjusting batch normalization, and finally sub-pixel convolution is performed to achieve upsampling to obtain super-resolution polarization images in the four polarization directions of 0°, 45°, 90° and 135°.

[0022] Furthermore, when training the super-resolution neural network model, the loss function is: ; Where N is the number of images, W is the image width, and H is the image height. and are the pixel values ​​of the super-resolution reconstructed image obtained by the super-resolution neural network model and the pixel values ​​of the high-resolution image in the image dataset, respectively. θ is the four polarization directions of 0°, 45°, 90° and 135°. The loss function weight of the linear polarization images with four different polarization directions is 0.25.

[0023] The beneficial effects of the method of the present invention are:

[0024] (1) Traditional super-resolution methods only input linear polarization images of a single polarization direction at a time, and do not use the pixel information of the other three polarization directions for super-resolution reconstruction. This method extracts the global features of the focal plane polarization image, and uses the spatial attention module to enhance the features of the same polarization direction, while retaining the features of the other polarization directions. The two types of features are used to complete the super-resolution reconstruction of four linear polarization images. According to the Stokes formula, in order to invert all polarization features, it is necessary to use images of different polarization directions to calculate Stokes parameters such as S1 and S2, and then perform polarization feature inversion; while the current polarization super-resolution algorithm directly inverts the polarization degree and polarization angle images, and the constructed loss function cannot invert parameters such as S1 and S2, and the polarization degree and polarization angle images cannot invert S1 and S2 information, resulting in the inability to perform full Stokes inversion. Secondly, there is relatively serious noise in the polarization angle image. It is difficult for the existing polarization super-resolution algorithm to correctly process the polarization angle image data set during training, resulting in an extremely low peak signal-to-noise ratio of the reconstructed polarization angle image, and the inability to restore the true polarization angle image of the target scene. The method described in the present invention directly reconstructs polarization images of different polarization directions and then calculates the polarization angle, so the super-resolution effect will not be affected by noise.

[0025] (2) The current sub-pixel convolution only reorganizes the feature map pixels. When performing super-resolution reconstruction of polarization images, the extracted feature map is related to the polarization direction. Only simple pixel reorganization does not fully utilize the features of different polarization directions, which affects the super-resolution effect of the polarization image. At the same time, the sub-pixel convolution needs to compress the number of channels of the feature map to the square of the magnification, which further leads to the loss of details of the extracted features. For this reason, the method described in the present invention proposes a channel-adaptive sub-pixel convolution, which can perform sub-pixel reconstruction on feature images with any number of channels, and at the same time, the channel weight is adaptively adjusted according to the proportion of feature information in different polarization directions, thereby improving the reconstruction quality of the polarization image.

[0026] The method of the present invention can be applied in the field of polarization detection imaging technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a method for super-resolution reconstruction of focal plane polarization images based on an attention mechanism in an embodiment of the present invention;

[0028] Figure 2 Schematic diagram of a super-resolution neural network model in an embodiment of the present invention;

[0029] Figure 3 Schematic diagram of channel adaptive sub-pixel convolution in an embodiment of the present invention;

[0030] Figure 4 The polarization image of the focal plane described in the embodiment of the present invention;

[0031] Figure 5 It is the 0° super-resolution polarization image described in the embodiment of the present invention;

[0032] Figure 6 The 45° super-resolution polarization image described in the embodiment of the present invention;

[0033] Figure 7 The 90° super-resolution polarization image described in the embodiment of the present invention;

[0034] Figure 8 This is the 135° super-resolution polarization image described in the embodiment of the present invention. DETAILED DESCRIPTION

[0035] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0036] Embodiment 1,

[0037] This embodiment provides a method for super-resolution reconstruction of focal plane polarization images based on an attention mechanism, the process is as follows: Figure 1 shown.

[0038] Step 1: Use the focal plane polarization detector to collect images. The resolution of the images collected by the focal plane polarization detector is 4000×3000. In order to improve the training speed of the model, the focal plane polarization images are cropped to 800×600. The collected images are split into linear polarization images with four polarization directions of 0°, 45°, 90° and 135°, with a resolution of 200×150. These four images are regarded as high-resolution images and used as the output data set of the super-resolution network model. Since the micro-polarizers with different polarization directions in the focal plane are staggered, the four split images are downsampled at intervals to simulate the decrease in spatial resolution caused by the focal plane. The resolution of the downsampled images is 100×75. The four downsampled images are recombined into low-resolution focal plane polarization images with a resolution of 200×150, which are used as the input data set of the super-resolution network model.

[0039] Step 2: Construct a super-resolution neural network model. The super-resolution network model is as follows: Figure 2As shown. The reconstructed low-resolution focal plane polarization image (200×150 focal plane polarization image) is used as input; the convolution layer is used to perform preliminary extraction of shallow features; the dense residual module is used to fully extract deep features; the spatial attention module is used to extract the attention map of the feature map and perform weighted fusion; the channel-adaptive sub-pixel convolution module is used to upsample the extracted feature map; four linear polarization images with a resolution of 200×150 and different polarization directions are obtained; the pixel loss function is used to calculate the difference between the high-resolution image reconstructed by the model and the high-resolution image in the output dataset.

[0040] The convolution layer for extracting shallow features consists of 96 5×5 convolution kernels, ensuring that the information of neighboring pixels with the same polarization direction as the pixel and the information of pixels with other polarization directions can be fully extracted at the same time.

[0041] There are 12 residual blocks in the dense residual module for extracting deep features. Each residual block consists of a 3×3 convolutional layer, a batch normalization layer, and a linear rectifier unit ReLU activation function. The residual connection is implemented by a 1×1 convolutional layer. The residual blocks are connected using 1×1 convolutional layers and average pooling layers to compress the number of channels of the feature map.

[0042] The spatial attention module is used to highlight the features of linear polarization images with different polarization directions. The feature maps are subjected to channel maximum pooling and channel average pooling to obtain two single-channel feature maps. The two single-channel feature images are combined to obtain a dual-channel feature map. The dual-channel feature map is convolved with a 5×5 convolution kernel and activated using a Sigmoid activation function to obtain an attention map composed of weight coefficients. The attention map is weightedly fused with each feature map to obtain a new feature map.

[0043] The channel adaptive sub-pixel convolution module is designed independently. First, the feature map obtained in the previous layer is subjected to maximum pooling and average pooling to obtain channel features of length c. The obtained channel features are input into a trainable channel adaptive sub-network to obtain channel weights. The channel adaptive sub-network consists of two convolutional layers and one fully connected layer, where the number of neurons in the fully connected layer is equal to the number of channels in the feature map. The weighted feature map is calculated according to the channel weight. Since four images with different polarization directions (0°, 45°, 90° and 135°) need to be reconstructed, conditional batch normalization is performed on the feature map. Different polarization directions are used as conditions to adjust the scaling factor and bias term of batch normalization, so as to realize the secondary dynamic adjustment of the feature channel weight, so that the corresponding output reconstructed image contains most of the features in the same polarization direction while retaining some features in other polarization directions, so as to improve the reconstruction quality of the polarization image. Finally, sub-pixel convolution is performed to achieve upsampling to obtain a super-resolution polarization image. Conditional batch normalization is performed in the sub-pixel reconstruction stage, so that the number of input channels of sub-pixel convolution is not limited by the magnification, and the detail information of the reconstructed image is improved. After reconstruction is completed, a super-resolution polarization image corresponding to the polarization direction is output.

[0044] Step 3: Train the above super-resolution neural network model.

[0045] All weights in the super-resolution neural network model are initialized with a mean of 0 and a normal distribution. The low-resolution (200×150) focal plane polarization image is used as input, and the convolution layer is used to extract shallow features, and the dense residual module is used to extract deep features. The spatial attention module is used to perform pixel-level discrimination on the feature map, and the features of the linear polarization image in the same polarization direction are enhanced, while a small amount of features of the linear polarization images in other polarization directions are retained. The final reconstructed super-resolution images of four different polarization directions are composed of all the information of the focal plane polarization image, rather than the traditional reconstruction of the linear polarization image in a single polarization direction. The feature map is upsampled using the channel-adaptive sub-pixel convolution module to complete the super-resolution reconstruction of the four linear polarization images in different polarization directions. The pixel absolute error between the reconstructed high-resolution image and the high-resolution image in the output data set is calculated as the pixel-level loss function, as shown in Equation (1).

[0046] (1)

[0047] Where N is the number of images, W is the image width, and H is the image height. and are the pixel values ​​of the super-resolution reconstructed image and the pixel values ​​of the high-resolution image in the image dataset, respectively. θ represents the four polarization directions of 0°, 45°, 90° and 135°. The weight of the loss function for the linear polarization images with four different polarization directions is 0.25.

[0048] The initial learning rate of the neural network model training is 0.001. To improve the convergence of the loss function, the learning rate decays exponentially per batch, with a decay rate of 0.975. The training lasts for 200 batches, and if the loss function reaches a low level at the end of 1 batch, the model can be saved immediately. Finally, as data enhancement, the input images are flipped horizontally and vertically during training, and randomly rotated by 0°, 90°, 180°, or 270°, so that the network model has stronger generalization, and the loss function is optimized using the Adam optimizer.

[0049] Step 4: Calculate the peak signal-to-noise ratio of the reconstructed super-resolution polarization image and the real high-resolution image as an average indicator to verify the effect of this method. Peak signal-to-noise ratio (PSNR) is usually expressed in decibels (dB). First, calculate the mean square error (MSE) of the two images. MSE is defined as:

[0050] (2)

[0051] in and Respectively represent the image at position Then use MSE to calculate PSNR, that is

[0052] (3)

[0053] in is the maximum possible pixel value of the image data, for an 8-bit image The value is 255. The larger the PSNR is, the closer the reconstructed image is to the real image. Figure 4 As shown, the four images of super-resolution reconstructed images are as follows Figure 5-8 As shown in the figure, the reconstructed PSNRs of the four polarization directions of 0°, 45°, 90°, and 135° are 39.11, 39.83, 39.64, and 39.32, respectively.

[0054] The method described in the present invention uses a focal plane polarization image as input, and reconstructs super-resolution polarization images of four polarization directions of 0°, 45°, 90° and 135° at one time, thereby improving the efficiency of super-resolution reconstruction, reducing the computing power overhead of repeated model training, and avoiding the problem of decreased adaptability of a single super-resolution reconstruction model when performing super-resolution reconstruction on linear polarization images of different polarization directions. The convolution layer and the dense residual module are used to extract shallow features and deep features respectively, and all the information of the focal plane polarization image is fully utilized. The spatial attention module is used to dynamically adjust the weights of features in different polarization directions, and the channel-adaptive sub-pixel convolution is used to increase the detail information of the reconstructed image, thereby improving the super-resolution reconstruction effect.

Claims

1. A method for super-resolution reconstruction of focal plane polarization images based on attention mechanism, characterized in that: The method comprises the following steps: S1, using a focal plane polarization detector to collect images and construct an image data set; S2. Constructing a super-resolution neural network model, in which a channel-adaptive sub-pixel convolution module is used to realize sub-pixel reconstruction of feature images of any number of channels, and at the same time, the channel weight is adaptively adjusted according to the proportion of feature information in different polarization directions; S3, using image datasets to train super-resolution neural network models; S4. Verify the super-resolution neural network model according to the evaluation indicators; S5. Use super-resolution neural network model to perform polarization image super-resolution reconstruction; The super-resolution neural network model includes an input layer, a convolution layer, a dense residual module, a spatial attention module, a channel adaptive sub-pixel convolution module and an output layer in sequence; The focal plane polarization image is used as input, the shallow features are extracted using the convolution layer, the deep features are extracted using the dense residual module, the attention map of the feature map is extracted using the spatial attention module and weighted fusion is performed, and the extracted feature map is upsampled using the channel adaptive sub-pixel convolution module to obtain four linear polarization images with different polarization directions; The spatial attention module performs channel maximum pooling and channel average pooling on the feature map to obtain two single-channel feature maps, combines the two single-channel feature maps to obtain a dual-channel feature map, performs convolution operation on the dual-channel feature map using a 5×5 convolution kernel and activates it using a Sigmoid activation function to obtain an attention map composed of weight coefficients, and performs weighted fusion of the attention map with each feature map to obtain a new feature map; The channel adaptive sub-pixel convolution module performs maximum pooling and average pooling on the feature map obtained in the previous layer to obtain channel features of length c, and inputs the obtained channel features into a channel adaptive sub-network to obtain channel weights. The channel adaptive sub-network is composed of two convolutional layers and a fully connected layer in sequence, wherein the number of neurons in the fully connected layer is equal to the number of channels of the feature map. A weighted feature map is obtained according to the channel weight calculation, and then the weighted feature map is subjected to conditional batch normalization. In the conditional batch normalization, four polarization directions of 0°, 45°, 90° and 135° are used as scaling factors and bias items for conditional adjustment of batch normalization. Finally, sub-pixel convolution is performed to achieve upsampling to obtain super-resolution polarization images of four polarization directions of 0°, 45°, 90° and 135°.

2. The method for super-resolution reconstruction of focal plane polarization images based on attention mechanism according to claim 1, characterized in that: Step S1 is specifically as follows: S11, using a focal plane polarization detector to collect images, split the images into linear polarization images with four polarization directions of 0°, 45°, 90° and 135°, and regard these four images as high-resolution images as the output of the super-resolution network model; S12, performing interval downsampling on the four split images, and recombining the four images after the interval downsampling into a component focal plane polarization image as an input of the super-resolution network model; S13. Construct an image data set including output data and corresponding input data according to the methods of step S11 and step S12 respectively.

3. The method for super-resolution reconstruction of focal plane polarization images based on attention mechanism according to claim 2, characterized in that: The convolutional layer consists of 96 5×5 convolution kernels.

4. The method for super-resolution reconstruction of focal plane polarization images based on attention mechanism according to claim 3, characterized in that: The dense residual module has a total of 12 residual blocks, each of which is composed of a 3×3 convolutional layer, a batch normalization layer, and a linear rectifier unit ReLU activation function. The residual connection is implemented by a 1×1 convolutional layer; the residual blocks are connected using a 1×1 convolutional layer and an average pooling layer.

5. The method for super-resolution reconstruction of focal plane polarization images based on attention mechanism according to claim 4, characterized in that: When training a super-resolution neural network model, the loss function is: ; Where N is the number of images, W is the image width, and H is the image height. and are the pixel values ​​of the super-resolution reconstructed image obtained by the super-resolution neural network model and the pixel values ​​of the high-resolution image in the image dataset, respectively. θ is the four polarization directions of 0°, 45°, 90° and 135°. The loss function weight of the linear polarization images with four different polarization directions is 0.

25.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images