A super-resolution reconstruction method of full polarization feature images based on deep learning

Abstract

A method for super-resolution reconstruction of full polarization feature images based on deep learning. It proposes a super-resolution reconstruction method that uses the linear polarization image and circular polarization image of the focal plane to output the light intensity image, polarization degree image, polarization angle image and ellipticity image at one time, which solves the technical problem that the resolution of the linear polarization image is reduced due to the focal plane imaging, which affects the accuracy of the full polarization feature inversion, and solves the technical problem that the inversion of the full polarization feature image requires additional image registration, which reduces the efficiency of super-resolution reconstruction. The method is specifically as follows: using a dual-channel co-aperture full polarization imaging system to collect images and construct an image data set; constructing a full polarization super-resolution neural network model; using the image data set to train the full polarization super-resolution neural network model; verifying the full polarization super-resolution neural network model according to the evaluation index; and using the full polarization super-resolution neural network model to perform super-resolution reconstruction of the full polarization feature image.

Description

Technical Field

[0001] The present invention belongs to the technical field of resolution reconstruction of full polarization feature images, and specifically relates to a full polarization feature image super-resolution reconstruction method based on deep learning. Background Art

[0002] Polarization imaging technology achieves target detection in complex environments by acquiring polarization information of different polarization directions of the target. Studies have shown that circularly polarized light has stronger penetration ability than linearly polarized light in scattering environments, and ellipticity images can better detect surface texture information of man-made objects such as metal targets. Compared with traditional linear polarization imaging systems, full polarization imaging can simultaneously acquire linear polarization information and circular polarization information of the target, and realize the inversion of all polarization characteristic information including full polarization degree, polarization angle, and ellipticity, and acquire target information from more dimensions, thereby improving target detection capabilities in different environments. The co-aperture full polarization imaging system based on focal plane and amplitude division can simultaneously capture a circular polarization image and a focal plane linear polarization image. The resolution of the linear polarization images with different polarization directions obtained by splitting the focal plane linear polarization image is 1 / 4 of that of the circular polarization image, making it impossible to perform "point-to-point" Stokes calculations between the polarization images, reducing the inversion accuracy of the polarization characteristic image.

[0003] Current image super-resolution algorithms usually perform super-resolution reconstruction based on the feature information of a single image; super-resolution algorithms for polarization images can only extract the feature information of the linear polarization image of the focal plane to perform super-resolution reconstruction of the intensity image, linear polarization degree image, and polarization angle image. According to the Stokes formula, the inversion of all polarization features needs to be based on both circular polarization information and linear polarization information. The single convolution structure used in existing super-resolution reconstruction algorithms cannot extract the feature information of circular polarization images and linear polarization images at the same time, and it is difficult to utilize the correlation between the feature information of different polarization images. In addition, when performing the inversion of the full polarization feature image, it is necessary to perform pixel-level alignment of the linear polarization image and the circular polarization image. The existing super-resolution method cannot avoid this link, which further reduces the efficiency of super-resolution reconstruction of the full polarization feature image. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a full polarization feature image super-resolution reconstruction method based on deep learning, and the method is specifically as follows:

[0005] S1. Use a dual-channel co-aperture full polarization imaging system to collect images and construct an image dataset;

[0006] S2. Construct a full polarization super-resolution neural network model; the full polarization super-resolution neural network model includes an input layer, a hybrid convolution layer, a channel attention module, a convolution layer, a feature selection layer, an upsampling reconstruction layer and an output layer in sequence;

[0007] S3, using the image dataset to train a full polarization super-resolution neural network model;

[0008] S4. Verify the full polarization super-resolution neural network model based on the evaluation indicators;

[0009] S5. Use the full polarization super-resolution neural network model to perform full polarization feature image super-resolution reconstruction.

[0010] Furthermore, the method for constructing the image dataset is specifically as follows:

[0011] S21, simultaneously collecting a focal plane linear polarization image and a circular polarization image with the same resolution in the same field of view, and registering the focal plane linear polarization image and the circular polarization image;

[0012] S22, splitting the acquired linear polarization image of the focal plane into linear polarization images with four different polarization directions of 0°, 45°, 90° and 135°;

[0013] S23, upsampling the linear polarization images with different polarization directions to obtain linear polarization images with the same resolution as the circular polarization image;

[0014] S24, four Stokes vectors S0, S1, S2 and S3 are obtained from the linear polarization images and circular polarization images of four different polarization directions of 0°, 45°, 90° and 135°, through:

[0015] ;

[0016] ;

[0017] ;

[0018] ;

[0019] Obtain, among which, , , , They correspond to the linear polarization images with four different polarization directions of 0°, 45°, 90° and 135° obtained by splitting the linear polarization image of the focal plane. is the circular polarization image;

[0020] S25, solving polarization characteristic images, wherein the polarization characteristic images include a full degree of polarization image DoP, a polarization angle image AoP, and an ellipticity image EoP, by:

[0021] ;

[0022] ;

[0023] ;

[0024] get;

[0025] S26, the polarization feature image and the light intensity image together constitute the output part of the image data set, the light intensity image is express;

[0026] S27, collecting the focal plane linear polarization images as a set of input parts in the image data set;

[0027] S28, the collected circular polarization images are used as another set of input parts in the image data set;

[0028] S29, repeat the method of step S21 to step S28 to construct an image data set including output data and corresponding input data.

[0029] Furthermore, the linear polarization image and the circular polarization image of the focal plane are simultaneously input into the full polarization super-resolution neural network model, and the hybrid convolution layer is used to perform feature extraction, and the independent features and common features of the two input images are respectively extracted; the channel attention module is used to calculate the weights of different feature maps and perform weighted operations; the convolution layer is used to further extract deep features; the feature selection layer is used to screen the features so that the corresponding polarization feature image is composed of correct feature reconstruction; the upsampling reconstruction layer is used to enlarge the size of the feature map, and output four high-resolution images including the intensity image, the full polarization degree image, the polarization angle image and the ellipticity image.

[0030] Furthermore, the hybrid convolution layer is composed of three different convolution kernels, wherein 64 5×5×1 convolution kernels are used to extract the features of the focal plane linear polarization image; 128 3×3×1 convolution kernels are used to extract the features of the circular polarization image; the focal plane linear polarization image and the circular polarization image are dimensionally superimposed to obtain a dual-channel image, and features are extracted from the dual-channel image using 128 3×3×2 convolution kernels; and three different types of feature images are obtained through hybrid convolution layer operations.

[0031] Furthermore, the channel attention module calculates the weights of different feature images, first calculates the channel attention between different feature images in the same type of feature images, performs maximum pooling and average pooling on the feature images in the spatial dimension, obtains two 1×1×c channel feature images, connects the channel feature images to obtain 1×1×2c channel feature images, and performs the above operations on three different types of feature images to obtain three types of 1×1×2c channel feature images respectively; uses a multilayer perceptron to learn the weights between channels of the three types of 1×1×2c channel feature images, the output layer of the multilayer perceptron has c neurons, and finally outputs a 1×1×c channel attention weight matrix, and performs weighted operations on the three different types of feature images.

[0032] Furthermore, the convolutional layer is used to further extract deep features.

[0033] Furthermore, the feature selection layer performs feature screening in two steps. First, the feature image corresponding to the linear polarization image is input into the channel of the reconstructed light intensity image and the polarization angle image, and the feature images corresponding to the linear polarization image, the circular polarization image, and the dual-channel image are input into the channel of the reconstructed total polarization degree image and the ellipticity image; then, according to the difference between the total polarization degree and the ellipticity, the feature images corresponding to the linear polarization image, the circular polarization image, and the dual-channel image are weighted predicted.

[0034] Furthermore, the upsampling reconstruction layer is composed of a transposed convolution kernel and a common convolution kernel, and the upsampling reconstruction layer outputs a reconstructed high-resolution intensity image, a high-resolution polarization angle image, a high-resolution total polarization degree image, and a high-resolution ellipticity image.

[0035] The beneficial effects of the method of the present invention are:

[0036] The currently existing polarization image super-resolution reconstruction algorithm only reconstructs the intensity image, linear polarization degree image, and polarization angle image through a focal plane linear polarization image, and is unable to invert the full polarization feature image, which affects the actual application effect of the full polarization imaging system. Compared with the linear polarization image, the full polarization feature image has a higher contrast, and the ellipticity image can better highlight the outline of the target according to the different surface textures and roughness of the target. The full polarization feature image super-resolution reconstruction method proposed in the present invention can enhance the application value of full polarization imaging. At the same time, a focal plane linear polarization image and a circular polarization image are input, and the intensity image, full polarization degree image, polarization angle image, and ellipticity image are output, so as to realize the super-resolution reconstruction of the full polarization feature image at one time.

[0037] The inversion of the full polarization feature image requires pixel-level registration of the circular polarization image and the linear polarization image. The existing super-resolution method requires additional image registration operations, which reduces the inversion efficiency of the full polarization image. The method of the present invention uses the circular polarization image and the focal plane linear polarization image that have completed pixel-level registration for inversion when constructing the output data set, so that the neural network model can learn the features of the registration image and reduce additional registration operations.

[0038] The single convolution structure used in the current image super-resolution algorithm can only extract features from a single image; when multiple images are input at one time, the differences and connections between different features of the multiple images are not fully utilized. The method described in the present invention designs a hybrid convolution layer including three convolution structures for the input focal plane polarization image and circular polarization image, respectively obtains feature information of the linear polarization image and the circular polarization image, and extracts common features of the two images at the same time, thereby overcoming the problem of low efficiency in utilizing features of multiple input images, thereby realizing super-resolution reconstruction of the full polarization feature image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flow chart of the full polarization feature image super-resolution reconstruction method based on deep learning in an embodiment of the present invention;

[0040] Figure 2 This is a structural diagram of the full polarization super-resolution neural network model in an embodiment of the present invention.

[0041] Figure 3 The polarization image of the focal plane described in the embodiment of the present invention;

[0042] Figure 4 is the circular polarization image described in the embodiment of the present invention;

[0043] Figure 5 Reconstructing a high-resolution light intensity image as described in an embodiment of the present invention;

[0044] Figure 6 Reconstructing a high-resolution full polarization image as described in an embodiment of the present invention;

[0045] Figure 7 Reconstructing a high-resolution polarization angle image as described in an embodiment of the present invention;

[0046] Figure 8 This is the reconstruction of a high-resolution polarization ellipticity image described in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0048] Embodiment 1,

[0049] This embodiment provides a full polarization feature image super-resolution reconstruction method based on deep learning, such as Figure 1 As shown, the method comprises the following steps:

[0050] S1. Use a dual-channel co-aperture full polarization imaging system to collect images and construct an image dataset;

[0051] S2. Construct a full polarization super-resolution neural network model; the full polarization super-resolution neural network model includes an input layer, a hybrid convolution layer, a channel attention module, a convolution layer, a feature selection layer, an upsampling reconstruction layer and an output layer in sequence;

[0052] S3, using the image dataset to train a full polarization super-resolution neural network model;

[0053] S4. Verify the full polarization super-resolution neural network model based on the evaluation indicators;

[0054] S5. Use the full polarization super-resolution neural network model to perform full polarization feature image super-resolution reconstruction.

[0055] Embodiment 2,

[0056] This embodiment further limits the embodiment 1 and further introduces step S1.

[0057] The dual-channel co-aperture full polarization imaging system is used to collect images and construct a dataset for training the full polarization super-resolution neural network model. The dual-channel co-aperture full polarization imaging system has a focal plane linear polarization detector and a circular polarization detector with the same resolution. The co-aperture optical system can simultaneously collect focal plane linear polarization images and circular polarization images with the same resolution in the same field of view, and align the focal plane linear polarization images and circular polarization images; the collected focal plane line images can be split into four linear polarization images with different polarization directions of 0°, 45°, 90° and 135°, and the split linear polarization images with different polarization directions are upsampled to make their resolution the same as the circular polarization image. According to the Stokes formulas (1)-(4), four Stokes vectors S0, S1, S2, and S3 are calculated, where S0 is the intensity image.

[0058] ; (1)

[0059] ; (2)

[0060] ; (3)

[0061] ; (4)

[0062] in , , , They correspond to the linear polarization images with four different polarization directions of 0°, 45°, 90° and 135° obtained by splitting the linear polarization image of the focal plane. is a circular polarization image. The above four Stokes vectors can be used to invert the three polarization characteristic images of degree of polarization (DoP), angle of polarization (AoP) and ellipticity of polarization (EoP). The calculation formulas are shown in equations (5)-(7).

[0063] ; (5)

[0064] ; (6)

[0065] ; (7)

[0066] The above polarization characteristic image and the light intensity image (the light intensity image is obtained by The collected linear polarization images of the focal plane serve as one of the input data sets of the super-resolution neural network model; the collected circular polarization images serve as another input data set of the super-resolution neural network model.

[0067] Embodiment 3,

[0068] This embodiment further limits the embodiment 1 and further introduces step S2.

[0069] Construct a full polarization super-resolution neural network model. The full polarization super-resolution neural network model is as follows Figure 2 shown.

[0070] The linear polarization image and circular polarization image of the focal plane are input into the input layer at the same time, and the hybrid convolution layer is used for feature extraction to extract the independent features and common features of the two input images respectively; the channel attention module is used to calculate the weights of different feature maps and perform weighted operations; the convolution layer is used to further extract deep features to improve the richness of detail information of the reconstructed polarization feature image; the feature selection layer is used to screen the features to ensure that the corresponding polarization feature image is composed of correct feature reconstructions; the upsampling reconstruction layer is used to enlarge the size of the feature map and output four high-resolution images including light intensity, total polarization degree, polarization angle and ellipticity.

[0071] The hybrid convolution layer is composed of three different convolution kernels to ensure that the feature information of the focal plane linear polarization image and the circular polarization image can be fully extracted. Among them, 64 5×5×1 convolution kernels are used to extract the features of the focal plane linear polarization image, which can cover more pixels in the same polarization direction; 128 3×3×1 convolution kernels are used to extract the feature information of the circular polarization image, and every two convolution kernels are grouped to ensure that the receptive field is the same as that of the 5×5×1 convolution kernel, while reducing the number of parameters of the model; the focal plane linear polarization image and the circular polarization image are dimensionally superimposed to obtain a dual-channel image, and 128 3×3×2 convolution kernels are used to extract its features, which can simultaneously extract the information of the focal plane linear polarization image and the circular polarization image.

[0072] After the mixed convolution layer operation, three different types of feature images can be obtained, and the channel attention module is used to calculate the weights of different feature images. First, the channel attention between different feature images in the same type of feature images is calculated, and the feature images are max-pooled and average-pooled in the spatial dimension to obtain two 1×1×c channel feature maps; the channel feature maps are connected to obtain a 1×1×2c channel feature map, and the weights between channels are learned using a multi-layer perceptron. The output layer of the multi-layer perceptron has c neurons, and finally a 1×1×c channel attention weight matrix is ​​output, and weighted operations are performed on the feature images.

[0073] The convolution layer is used to further extract deep features and the feature selection layer is used for feature selection. The feature selection layer is divided into two steps. First, the feature map is selected according to the different output images. From equations (1)-(7), it can be seen that the intensity image and the polarization angle image are only related to the linear polarization information. Therefore, before upsampling and reconstruction, the linear polarization features extracted by the hybrid convolution layer are input into the channels of the reconstructed intensity image and the polarization angle image to avoid interference from the circular polarization features. For the full polarization degree image and ellipticity image that require all linear polarization information and circular polarization information, all features are input into the corresponding resolution reconstruction channel. Then, according to the difference between the full polarization degree and the ellipticity, the weights of the three different feature images mentioned above are predicted so that the reconstructed image contains different feature information. Finally, the resolution is reconstructed by the upper reconstruction layer composed of the transposed convolution kernel and the ordinary convolution kernel, and the corresponding high-resolution polarization feature image is output.

[0074] Embodiment 4,

[0075] This embodiment further limits the embodiment 1 and further introduces step S3.

[0076] The above full polarization super-resolution neural network model is trained.

[0077] The parameters of the neural network model are initialized, and a circular polarization image and a simulated focal plane linear polarization image are input at the same time. The hybrid convolution layer is used to extract three different types of feature information, namely, linear polarization image features, circular polarization image features, and mixed features of the two images. The channel attention module is used to predict the weight of each type of feature map and perform weighted operations. The convolution layer is used to further extract the deep features of each weighted feature map. The feature selection layer is used to filter the information required by different polarization feature images. The feature weights of the full polarization image and ellipticity image with multiple types of feature information are predicted. The transposed convolution kernel and the ordinary convolution kernel are used for reconstruction, and high-resolution intensity image, full polarization degree image, polarization angle image, and ellipticity image are output respectively. The error between the reconstructed high-resolution polarization feature image and the real image is calculated using a custom loss function. The loss function is shown in formula (8).

[0078] (8)

[0079] Where N is the number of images, W is the image width, and H is the image height. , , , , , , , The pixel values ​​of the reconstructed high-resolution polarization feature image corresponding to the light intensity, total polarization degree, polarization angle and ellipticity are the pixel values ​​of the high-resolution image in the high-resolution output data set of the neural network model. Here, λ1, λ2, λ3, λ4 are the weight coefficients of the loss function, which are set to 1, 1, 0.2, and 0.2 respectively according to experience. The initial learning rate of the neural network model training is 0.001. In order to improve the convergence of the loss function, the learning rate decays exponentially per batch, and the decay rate is 0.99. The training lasts for 150 batches. As data enhancement, the input image is randomly rotated and flipped during the training process to improve the generalization ability of the model. The Adam optimizer is used to optimize the loss function.

[0080] Embodiment 5,

[0081] This embodiment further limits the embodiment 1 and further introduces step S4.

[0082] Calculate the structural similarity between the reconstructed full polarization feature image and the real image to evaluate the reconstruction result. The calculation formula is as follows

[0083] ; (9)

[0084] in and represent the reconstructed image and the real image respectively, is the average grayscale of the image as a brightness estimate, is the standard deviation of the image as a contrast estimate, and is the structural similarity constant, and the value of structural similarity is between 0 and 1. The larger the value, the smaller the image distortion and the better the super-resolution reconstruction effect. The trained full polarization super-resolution neural network model is used for testing. The input focal plane linear polarization image and circular polarization image are as follows: Figure 3 Figure 4 As shown, the reconstructed polarization feature image is as follows Figure 5-Figure 8 As shown in Figure 2, the structural similarities of the four images are 0.97, 0.89, 0.63, and 0.72 respectively.

Claims

1. A full polarization feature image super-resolution reconstruction method based on deep learning, characterized in that: The method comprises the following steps: S1. Use a dual-channel co-aperture full polarization imaging system to collect images and construct an image dataset; S2. Construct a full polarization super-resolution neural network model; the full polarization super-resolution neural network model includes an input layer, a hybrid convolution layer, a channel attention module, a convolution layer, a feature selection layer, an upsampling reconstruction layer and an output layer in sequence; The hybrid convolution layer is composed of three different convolution kernels, which are respectively used to extract the features of the focal plane linear polarization image, the features of the circular polarization image, and the features of the dual-channel image obtained by dimensional superposition of the focal plane linear polarization image and the circular polarization image; S3, using the image dataset to train a full polarization super-resolution neural network model; S4. Verify the full polarization super-resolution neural network model based on the evaluation indicators; S5. Perform full polarization feature image super-resolution reconstruction using a full polarization super-resolution neural network model; the method for constructing the image data set is specifically as follows: S21, simultaneously collecting a focal plane linear polarization image and a circular polarization image with the same resolution in the same field of view, and registering the focal plane linear polarization image and the circular polarization image; S22, splitting the acquired linear polarization image of the focal plane into linear polarization images with four different polarization directions of 0°, 45°, 90° and 135°; S23, upsampling the linear polarization images with different polarization directions to obtain linear polarization images with the same resolution as the circular polarization image; S24, four Stokes vectors S0, S1, S2 and S3 are obtained from the linear polarization images and circular polarization images of four different polarization directions of 0°, 45°, 90° and 135°, through: ； ； ； ； Obtain, among which, , , , They correspond to the linear polarization images with four different polarization directions of 0°, 45°, 90° and 135° obtained by splitting the linear polarization image of the focal plane. is the circular polarization image; S25, solving polarization characteristic images, wherein the polarization characteristic images include a full degree of polarization image DoP, a polarization angle image AoP, and an ellipticity image EoP, by: ； ； ； get; S26, the polarization feature image and the light intensity image together constitute the output part of the image data set, the light intensity image is express; S27, collecting the focal plane linear polarization images as a set of input parts in the image data set; S28, the collected circular polarization images are used as another set of input parts in the image data set; S29, repeat the method of step S21 to step S28 to construct an image data set including output data and corresponding input data.

2. The method for super-resolution reconstruction of full polarization feature images based on deep learning according to claim 1, characterized in that: The linear polarization image and the circular polarization image of the focal plane are simultaneously input into the full polarization super-resolution neural network model, and the hybrid convolution layer is used to extract features, and the independent features and common features of the two input images are respectively extracted; the channel attention module is used to calculate the weights of different feature maps and perform weighted operations; and the convolution layer is used to further extract deep features; The feature selection layer is used to screen the features so that the corresponding polarization feature image is composed of correct feature reconstruction; the upsampling reconstruction layer is used to enlarge the size of the feature map and output four high-resolution images including the intensity image, the full polarization degree image, the polarization angle image and the ellipticity image.

3. The method for super-resolution reconstruction of full polarization feature images based on deep learning according to claim 2, characterized in that: In the hybrid convolution layer, 64 5×5×1 convolution kernels are used to extract the features of the linear polarization image of the focal plane; 128 3×3×1 convolution kernels are used to extract the features of the circular polarization image; 128 3×3×2 convolution kernels are used to extract the features of the dual-channel image; three different types of feature images are obtained after the hybrid convolution layer operation.

4. The method for super-resolution reconstruction of full polarization feature images based on deep learning according to claim 3, characterized in that: The channel attention module calculates the weights of different feature images. First, the channel attention between different feature images in the same type of feature images is calculated, and the feature images are respectively subjected to maximum pooling and average pooling in the spatial dimension to obtain two 1×1×c channel feature images. The channel feature images are connected to obtain 1×1×2c channel feature images. The above operation is performed on three different types of feature images to obtain three types of 1×1×2c channel feature images respectively. The weights between channels of the three types of 1×1×2c channel feature images are learned using a multilayer perceptron, and the output layer of the multilayer perceptron has c neurons. Finally, a 1×1×c channel attention weight matrix is ​​output, and weighted operations are performed on the three different types of feature images.

5. The method for super-resolution reconstruction of full polarization feature images based on deep learning according to claim 4, characterized in that: The feature selection layer performs feature screening in two steps. First, the feature image corresponding to the linear polarization image is input into the channel of the reconstructed light intensity image and polarization angle image, and the feature images corresponding to the linear polarization image, circular polarization image, and dual-channel image are input into the channel of the reconstructed total polarization degree image and ellipticity image; then, according to the difference between the total polarization degree and the ellipticity, the weights of the feature images corresponding to the linear polarization image, circular polarization image, and dual-channel image are predicted.

6. The method for super-resolution reconstruction of full polarization feature images based on deep learning according to claim 5, characterized in that: The upsampling reconstruction layer is composed of a transposed convolution kernel and a common convolution kernel, and the upsampling reconstruction layer outputs a reconstructed high-resolution light intensity image, a high-resolution polarization angle image, a high-resolution total polarization degree image, and a high-resolution ellipticity image.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images