Compressive Sensing Polarization Super-Resolution Imaging Method Based on Deep Learning Framework

Abstract

A compressive sensing polarization super-resolution imaging method based on a deep learning framework, which relates to the fields of computational imaging and polarization detection technology. This method includes the following three steps: compressive coding sampling, super-resolution reconstruction, and depolarization demosaicing; among which, the super-resolution reconstruction and depolarization demosaicing stages are realized by a two-layer generative adversarial network; the reconstruction of the super-resolution polarization image is realized by using the image obtained by a low-resolution defocused plane polarization detector at a lower sampling rate, and S0, DOLP, and AOP images with high resolution and no polarization demosaicing are generated. Based on depolarization demosaicing, the resolution of the S0, DOLP, and AOP images generated by this method is much higher than the original resolution of the polarization detector; a mapping relationship between the original polarization image and the S0, DOLP, and AOP polarization characteristic images is established by using a deep learning network, and high-resolution S0, DOLP, and AOP images without demosaicing are directly generated from the original polarization demosaicing image. The present invention can effectively avoid secondary errors.

Description

Technical Field

[0001] The present invention relates to the technical fields of computational imaging and polarization detection, and particularly relates to a polarization super-resolution reconstruction method based on deep learning and compressive sensing. Background Art

[0002] Polarization imaging technology is applicable to imaging low-contrast targets in a variety of complex environments. In addition to being widely used in the military field, it also has great potential in industrial inspection, agricultural supervision, biomedicine, atmospheric monitoring, public security reconnaissance, etc.

[0003] The sub-focal plane polarization detector is an emerging technology in the field of polarization imaging. It integrates four sub-wavelength metal gratings with different polarization directions in four adjacent pixels to form a superpixel. Since the sub-focal plane polarization detector can simultaneously collect multiple polarization angle information, and has a simple structure, high integration, stable performance, and good real-time performance, it has been widely used in the field of polarization detection. However, different polarization directions are distributed on adjacent pixels of the sub-focal plane polarization detector, resulting in a mosaic-like polarization image being collected, which leads to a loss of polarization imaging resolution and a lack of instantaneous field of view. Therefore, in the field of polarization imaging, how to remove the mosaic effect in sub-focal plane polarization imaging has always been the focus of research.

[0004] The theory of compressive sensing breaks through the limitation of the Nyquist sampling frequency for signal sampling, simultaneously acquires and compresses the signal at a lower sampling rate, and reconstructs a high-dimensional signal. Using compressive sensing technology, a low-resolution detector can achieve high-resolution detection under the limited system bandwidth. Combining with the powerful learning ability of deep learning, the mapping relationship between low-resolution images and high-resolution images is obtained through training with a large amount of data, thereby realizing super-resolution imaging. Therefore, the super-resolution technology combining compressive sensing and deep learning has great advantages in improving the resolution of sub-focal plane polarization imaging.

[0005] In 2020, Garrett C. Sargent [Garrett C. Sargent, Bradley M. Ratliff, and Vijayan K. Asari, "Conditional generative adversarial network demosaicing strategy for division of focal plane polarimeters," Opt. Express 28, 38419-38443 (2020)] et al. proposed using a conditional generative adversarial network (cGAN) to remove the mosaic of polarized images. Among them, the U-Net architecture is added to the generator, and the PatchGAN architecture is added to the discriminator. This network structure improves the generation probability of high-frequency content and can finally output a full-resolution polarized feature image. This method uses deep learning to restore the problem of resolution loss in division of focal plane polarization imaging, making the resolution of the generated image consistent with that of the division of focal plane polarization detector. However, such methods cannot further improve the resolution of polarization imaging, and still cannot meet the high-resolution imaging requirements for detection spectral bands such as infrared. Summary of the Invention

[0006] To solve the problem in the prior art that the resolution of polarization imaging cannot be further improved and the high-resolution imaging requirements for detection spectral bands such as infrared cannot be met, the present invention provides a compressive sensing polarization super-resolution imaging method based on a deep learning framework, which can simultaneously achieve super-resolution reconstruction with a resolution higher than that of the original division of focal plane polarization detector on the basis of removing the mosaic of the division of focal plane polarization camera.

[0007] To solve the problems existing in the prior art, the technical solution provided by the present invention is as follows:

[0008] A compressive sensing polarization super-resolution imaging method based on a deep learning framework, the method includes the following three steps: compressive coding sampling, super-resolution reconstruction, and depolarization demosaicing; among them, the super-resolution reconstruction and depolarization demosaicing stages are implemented by a two-layer generative adversarial network;

[0009] First step, compressive coding sampling: using a digital micromirror device array DMD to perform compressive sensing coding measurement sampling on the imaging scene; first project the imaging scene onto the DMD, perform independent block coding on the imaging scene through the DMD, and each block area after division is imaged on a pixel of the division of focal plane polarization detector respectively, and the division of focal plane polarization detector collects the encoded image information to generate an original polarized mosaic image I (mos，k) ;

[0010] Step 2, super-resolution reconstruction: The polarization directions between adjacent pixels in the original polarization mosaic image are different. The pixels with polarization degrees of 0°, 45°, 90°, and 135° are segmented as a super-pixel, and then recombined into a one-dimensional measurement value vector y Ki ; and the corresponding coding mask is recombined into a measurement matrix Φ for reconstruction; the measurement vector y Ki and the measurement matrix Φ are used as input data and fed into the generator G0 of the first-layer generative adversarial network for compressive sensing reconstruction to generate a pseudo high-resolution image I HRmos , and the generated pseudo-image and the real high-resolution polarization image are discriminated for authenticity in the discriminator D0, and the network parameters in G0 and D0 are iteratively updated according to the loss function to make the generated image continuously approach the real high-resolution polarization image;

[0011] Step 3, depolarization of the mosaic: The image I generated by G0 in the second stage HRmos is input into the generator G1 of the second-layer generative adversarial network to establish a mapping relationship between the polarization mosaic image and the polarization characteristics, so as to generate the S0, DOLP, and AOP images after depolarization of the mosaic. The generated S0, DOLP, and AOP images and the real S0, DOLP, and AOP images are evaluated in the discriminator D1, and the network parameters in G1 and D1 are iteratively updated according to the loss function to make the generated image continuously approach the real high-resolution S0, DOLP, and AOP images.

[0012] The beneficial effects of the present invention are as follows:

[0013] 1. The present invention combines the compressive sensing theory with the generative adversarial network to form a polarization super-resolution imaging method based on deep learning and compressive sensing. This method uses the image obtained by the low-resolution focal plane polarization detector at a lower sampling rate to realize the reconstruction of the super-resolution polarization image, and generates high-resolution S0, DOLP, and AOP images without polarization mosaic. Different from the previous deep learning methods for depolarizing imaging mosaics, based on depolarizing the mosaic, the S0, DOLP, and AOP images generated by this method have a resolution much higher than the original resolution of the polarization detector.

[0014] 2. The ultimate goal of polarization imaging is to obtain a polarization characteristic image. In the past, polarization imaging used the original low-resolution polarization image to reconstruct high-resolution intensity images in different polarization directions, and then calculated the polarization characteristic image according to the formula. However, the present invention uses a deep learning network to establish a mapping relationship between the original polarization image and the polarization characteristic images of S0, DOLP, and AOP, and directly generates high-resolution mosaic-free S0, DOLP, and AOP images from the original polarization mosaic image. This process not only reduces the training task, but also effectively avoids secondary errors because the reconstructed polarization direction image itself also has reconstruction errors, and the present invention does not require the process of calculating the polarization characteristic image again from the reconstructed polarization direction image. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flowchart of the compressive sensing polarization super-resolution imaging method based on a deep learning framework according to the present invention;

[0016] Figure 2 It is a schematic diagram of DMD encoded sampling principle, where 1 is the target scene, 2 is the telescopic objective lens, 3 is the digital micromirror array DMD, 4 is the projection lens, and 5 is the split focal plane polarization detector;

[0017] Figure 3 It is a schematic diagram of measurement value recombination, where a is the encoded mask loaded on the DMD, b is the area corresponding to the super pixel of the DMD micromirror and the split focal plane polarization detector, c is the vector form of the encoded mask a, d is the result collected by the polarization detector, e is the recombined column vector group of the encoded vector c, and f is the vector form of the area b corresponding to the pixel of the DMD micromirror and the split focal plane polarization detector;

[0018] Figure 4 It is Figure 2 the micro-polarization array arrangement of the split focal plane polarization detector 5 in

[0019] Figure 5 It is the structure diagram of the generator G0 of the first layer of adversarial generation network;

[0020] Figure 6 It is the structure diagram of the discriminator D0 of the first layer of adversarial generation network;

[0021] Figure 7 It is the structure diagram of the generator G1 of the second layer of adversarial generation network;

[0022] Figure 8 It is the structure diagram of the residual module;

[0023] Figure 9 It is the structure diagram of the discriminator D1 of the second layer of adversarial generation network;

[0024] Figure 10The original polarization image I captured by the polarization detector (mos，k) ;

[0025] Figure 11 The high-resolution polarization mosaic image I generated by the first-layer generative adversarial network HRmos ;

[0026] Figure 12 For using Figure 9 The original polarization image I (mos，k) The S0 image solved;

[0027] Figure 13 For using Figure 10 The high-resolution polarization mosaic image I HRmos The S0 image solved;

[0028] Figure 14 For using Figure 9 The original polarization image I (mos，k) The DOLP image solved;

[0029] Figure 15 For using Figure 10 The high-resolution polarization mosaic image I HRmos The DOLP image solved;

[0030] Figure 16 Using Figure 9 The original polarization image I (mos，k) The AOP image solved;

[0031] Figure 17 For using Figure 10 The high-resolution polarization mosaic image I HRmos The AOP image solved. Detailed implementation manners

[0032] The present invention will be further described in detail below with reference to the accompanying drawings.

[0033] The specific steps for implementing the compressive sensing polarization super-resolution imaging method based on the deep learning framework are as follows:

[0034] The first step, compressive coding sampling, includes the following steps:

[0035] Step 1, the digital micromirror device array DMD encodes the imaging scene. First, project the imaging scene onto the DMD, and perform independent block coding on the imaging scene through the DMD. Each block area after partitioning is imaged on a pixel of the sub-focal plane polarization detector, and the compressed sampled polarization mosaic original image I (mos，k) is obtained. The number of the obtained polarization mosaic original images is equal to the number of coding measurements T, and the kth measurement is represented by k.

[0036] Step 2: Obtain the training set data of the reconstructed image:

[0037] 2.1. Original polarization mosaic image I (mos，k) The polarization degrees between adjacent pixels are different. The obtained original polarization mosaic image I (mos，k) is segmented with pixels having polarization degrees of 0°, 45°, 90°, and 135° as a superpixel region, and the value of the superpixel is converted into a one-dimensional measurement value vector y i , where y i represents the i-th superpixel; at the same time, the corresponding coded mask matrix is reorganized to generate a coded matrix M for reconstruction i .

[0038] 2.2. Combine the T superpixel measurement value vectors y i at the same position on different polarization coded images into a vector y Ki , and combine the coded matrix M i into a measurement matrix Φ. y Ki and Φ are used as the input data of the generator.

[0039] The second step, super-resolution reconstruction, includes the following steps:

[0040] Step 1: Input the measurement vector y Ki and the measurement matrix Φ into the generator G0 of the first layer of the generative adversarial network. Perform a linear mapping on y Ki through a fully connected layer to obtain an initial reconstruction result After that reconstruct and generate an output result after passing through four convolutional layers

[0041]

[0042] Among them, the measurement value y Ki and the measurement matrix Φ perform preliminary reconstruction on the image block of the superpixel region by using a fully connected network.

[0043] The expression is

[0044] In the expression represents the initial reconstruction result, W p represents the weight parameter matrix of the fully connected layer, and b i represents the bias.

[0045] The generator G0 consists of a fully-connected layer and four convolutional layers. The fully-connected layer is used to perform preliminary reconstruction on the compressed sampling information; the first two convolutional layers extract features from the image, the third convolutional layer is a sub-pixel convolutional layer to enhance the resolution of the image, and the fourth convolutional layer outputs the reconstruction result. The activation function of the first three convolutional layers is ReLU, and the activation function of the fourth layer is the hyperbolic tangent function tanh.

[0046] Step 2: Input the reconstruction result of Step 1 and the real image patch into the discriminator D0 of the first-layer generative adversarial network for discrimination. The discriminator D0 has 4 convolutional layers. The activation function of the first three convolutional layers is Leaky-ReLU. The result of the fourth layer is used as the output of the discriminator D0, and the activation function is the Sigmoid function.

[0047] Step 3: Calculate the loss functions of the generator G0 and the discriminator D0, and use the calculated loss to alternately reconstruct and train the generator G0 and the discriminator D0 using the Adam optimizer, while updating the parameters inside the generator G0 and the discriminator D0.

[0048] Among them, the adversarial loss of the generator G0 is:

[0049]

[0050] The content loss of the generator G0 is:

[0051]

[0052] The total loss function of the generator G0 in the formula is:

[0053]

[0054] The loss function of the discriminator D0 is:

[0055]

[0056] In the formula, φ is the pre-trained VGG-16, is the low-resolution image, is the high-resolution image, and C, H, and W represent the number of channels, the length of the image, and the width of the image respectively.

[0057] Step 4: When the error of the training result reaches the set error, the reconstruction iterative training is completed, and the generator G0 and the discriminator D0 are saved.

[0058] Step 5: Stitch the reconstructed image patches into a complete polarization imaging scene image I HRmos , and at this time I HRmos is a high-resolution polarization intensity image with a polarization mosaic.

[0059] The third step is to remove the polarization mosaic, including the following steps:

[0060] Step 1: Input the high-resolution polarization mosaic intensity image I HRmos into the generator G1 in the second-layer generative adversarial network. The polarization directions of adjacent pixels are all different, and each superpixel contains polarization degrees of 0°, 45°, 90°, and 135°. Using the relationship between the intensity information of different polarization directions in I HRmos and S0, DOLP, and AOP, establish the mapping relationship between the high-resolution polarization mosaic intensity image I HRmos and the high-resolution polarization characteristic image, and generate the polarization characteristic images I HRmos of S0, DOLP, and AOP, I S0 、I DOLP 、I AOP .

[0061] Among them, G1 contains two convolutional layers and two residual modules. One convolutional layer is used for preliminary feature extraction, and one convolutional layer is used for result output. The activation function of the convolutional layer is ReLU, and the activation function of the output convolutional layer is the hyperbolic tangent function tanh.

[0062] The relationship between the intensity information of different polarization directions in I HRmos and S0, DOLP, and AOP is

[0063]

[0064]

[0065]

[0066] Step 2: Input the generation result of Step 1 and the polarization characteristic images of real S0, DOLP, and AOP into the discriminator D1 in the second-layer generative adversarial network for discrimination. The discriminator D1 consists of 5 convolutional layers. The activation function of the first four convolutional layers is Leaky-ReLU. The activation function of the fifth convolutional layer is the Sigmoid function.

[0067] Step 3: Calculate the loss functions of the generator G1 and the discriminator D1, and use the calculated loss to alternately reconstruct and train the generator G1 and the discriminator D1 using the Adam optimizer function, and update the internal parameters of the generator G1 and the discriminator D1 at the same time.

[0068] Among them, the adversarial loss function of the generator G1 is:

[0069]

[0070] The L1 smoothing loss function of the generator G1 is:

[0071] L smooth-L1 = smooth L1 (H - G1(I HRmos )) (10)

[0072]

[0073] The pixel loss function of the generator G1 is:

[0074]

[0075] The total loss function of the generator G1 in the formula is:

[0076]

[0077] The loss function of the discriminator D1 is:

[0078]

[0079] In the formula, d represents the gradient of the image, and H is the expected high - resolution non - mosaic polarized image.

[0080] Step 4: When the error of the training result reaches the set error, the reconstruction iterative training is completed, the final reconstruction result is output, and the generator G1 and the discriminator D1 are saved.

[0081] Example:

[0082] As Figure 1 shown, the polarization super - resolution imaging method based on deep learning and compressive sensing has the following process steps:

[0083] The first step is compressive encoding sampling, including the following steps:

[0084] Step 1: The digital micromirror device array DMD encodes the imaging scene, and the compressive - sampled polarized mosaic original image I (mos，k) is obtained through the defocused - plane polarization camera. The number of obtained polarized mosaic original images is equal to the number of encoding measurement times T, and the k - th measurement is represented by k. In this example, the size of the DMD is taken as 1920×1080 for illustration, and the resolution of the polarization defocused - plane detector is 960×540.

[0085] Among them, the DMD performs compressive sensing encoding measurement sampling on the imaging scene, as Figure 2As shown in the figure, the polarization imaging system consists of a target scene 1, a telescopic objective lens 2, a digital micromirror device (DMD) 3, a projection lens 4, and a split focal plane polarization detector 5. First, the target scene 1 is projected onto the 1920×1080 micromirrors of the digital micromirror device (DMD) 3. By loading a random coding matrix, each micromirror on the digital micromirror device (DMD) 3 is controlled to achieve independent block coding of the imaging scene. Then, the split focal plane polarization detector 5 collects the encoded image information to generate the original polarization mosaic image I (mos，k) 。

[0086] Step 2: Obtain the training set data of the reconstructed image:

[0087] 2.1. The measurement value recombination process is as Figure 3 shown. The encoded mask a loaded on the digital micromirror device (DMD) 3 is converted into a one-dimensional column vector c. The area b corresponding to the micromirrors of the digital micromirror device (DMD) 3 and the pixels of the polarization detector shows a 4×4 micromirror area. Every 2×2 micromirrors are each imaged on one pixel of the split focal plane polarization detector 5. The 4×4 micromirror area is projected onto a super-pixel with a size of 2×2 on the split focal plane polarization detector. One super-pixel contains polarization information with polarization degrees of 0°, 45°, 90°, and 135°.

[0088] During the measurement value recombination process, the vector form c of the encoded mask a is rearranged into a vector group e according to the correspondence between the DMD micromirrors and the pixels of the polarization detector, and the vector group e is formed into an encoding matrix M for reconstruction i 。 The vector form f of the area b corresponding to the DMD micromirrors and the pixels of the polarization detector is multiplied by the encoding matrix M i to obtain the result d of polarization detection acquisition. The polarization degrees between adjacent pixels in the original polarization mosaic image I (mos，k) are different. As Figure 4 shown, the obtained original polarization mosaic image I (mos，k) is segmented according to the pixels with polarization degrees of 0°, 45°, 90°, and 135° as one super-pixel area. The result d of polarization detection acquisition is in the vector form of a super-pixel, and the super-pixel is converted into a one-dimensional measurement value vector y i 。 y i represents the i-th super-pixel, where the total number is I = 480 * 270.

[0089] 2.2. Combine the measurement value vectors y i of T super-pixels at the same position on different polarization encoded images into a vector y Ki , and combine the encoding matrix M i into a measurement matrix Φ. y KiUsing ξ and Φ as the input data of the generator, during training, the sampling times are selected as 8, 4, and 1, and the corresponding sampling rates are 0.5, 0.25, and 0.0625 respectively.

[0090] Step 2, super-resolution reconstruction, includes the following steps:

[0091] Step 1: Input the measurement vector y Ki and Φ into the generator G0 of the first-layer generative adversarial network. Perform a linear mapping on y Ki through a fully connected layer to obtain the initial reconstruction result After that After passing through four convolutional layers, the output result is reconstructed and generated The measured value y Ki and the measurement matrix Φ are used to perform a preliminary reconstruction of the image patch, which is implemented using a fully connected network.

[0092] The expression is

[0093] where represents the initial reconstruction result, W p represents the weight parameter matrix of the fully connected layer, and b i represents the bias.

[0094] As Figure 5 shown, the generator G0 consists of a fully connected layer and four convolutional layers. The fully connected layer is used to perform a preliminary reconstruction of the compressive sampling information. The first convolutional layer has 64 convolutional kernels of size 3×3. The activation function of the first layer is ReLU, and the sampling stride is 1. The second convolutional layer has 32 convolutional kernels of size 2×2. The activation function of the second layer is ReLU, and the sampling stride is 2. The third layer is a sub-pixel convolutional layer with 16 convolutional kernels of size 2×2. The activation function of the third layer is ReLU, and the sampling stride is 2. The fourth convolutional layer has 8 convolutional kernels of size 1×1, and the sampling stride is 1. The result of the fourth layer is used as the output of the generator G0, and the activation function is the hyperbolic tangent function tanh. The outputs of all convolutional layers are respectively processed by a batch normalization layer and then input into the activation function, and appropriate zero-padding is used for all convolutional layers.

[0095] Step 2: Input the reconstruction result of Step 1 and the real image patch into the discriminator D0 of the first-layer generative adversarial network for evaluation. As Figure 6As shown, the discriminator D0 has 4 convolutional layers. The first convolutional layer has 1 convolutional kernel of size 4×4, the activation function of the first layer is Leaky-LURe, and the sampling stride is 1. The second convolutional layer has 16 convolutional kernels of size 3×3, the activation function of the second layer is Leaky-ReLU, and the sampling stride is 1. The third convolutional layer has 32 convolutional kernels of size 2×2, the activation function of the third layer is Leaky-ReLU, and the sampling stride is 1. The fourth convolutional layer has 64 convolutional kernels of size 2×2, and the result of the fourth layer is used as the output of the discriminator D0. The activation function is the Sigmoid function, and the sampling stride is 1. The outputs of all convolutional layers are respectively processed by a batch normalization layer and then input into the activation function. Appropriate zero-padding is used for all convolutional layers.

[0096] Step 3: Calculate the loss functions of the generator G0 and the discriminator D0, and perform alternating reconstruction training on the generator G0 and the discriminator D0 respectively with the Adam optimizer according to the calculated loss functions, while updating the parameters inside the generator G0 and the discriminator D0.

[0097] Among them, the adversarial loss of the generator G0 is:

[0098]

[0099] The content loss of the generator G0 is:

[0100]

[0101] The total loss function of the generator G0 in the formula is:

[0102]

[0103] The loss function of the discriminator D0 is:

[0104]

[0105] In the formula, φ is the pre-trained VGG-16, is the low-resolution image, is the high-resolution image, and C, H, and W respectively represent the number of channels, the length of the image, and the width of the image.

[0106] Step 4: When the error of the training result reaches the set error, the reconstruction iterative training is completed, and the generator G0 and the discriminator D0 are saved.

[0107] Step 5: Stitch the I image patches of size 4×4 after reconstruction into a complete polarization imaging scene image I with a resolution of 1920×1080 HRmos . As Figure 11 shown, at this time I HRmos is a high-resolution intensity image with a polarization mosaic.

[0108] The third step is to remove the polarization mosaic, including the following steps:

[0109] Step 1: Input the high-resolution polarization mosaic intensity image I HRmos into the generator G1 in the second-layer generative adversarial network. The polarization directions of adjacent pixels are all different, and each superpixel contains polarization degrees of 0°, 45°, 90°, and 135°. Using the relationship between the intensity information of different polarization directions in I HRmos and S0, DOLP, and AOP, establish the mapping relationship between the high-resolution polarization mosaic intensity image I HRmos and the high-resolution polarization characteristic image, and generate the polarization characteristic images I HRmos of S0, DOLP, and AOP, I S0 、I DOLP 、I AOP .

[0110] As Figure 7 shown, the generator G1 has 2 convolutional layers and 2 residual modules. The first convolutional layer has 256 convolutional kernels of size 3×3. The activation function of the first layer is ReLU, and the sampling stride is 1. The second layer is a residual module mainly for feature extraction. The third layer is a residual module mainly for feature mapping. The residual block structure is as Figure 8 shown. The fourth layer is a convolutional layer containing 16 convolutional kernels of size 3×3. The result of the fourth layer is used as the output of the generator G1, and the activation function is the hyperbolic tangent function tanh. The outputs of all convolutional layers are respectively processed by batch normalization layers and then input into the activation function. In order to keep the sizes of the intermediate feature maps in the residual reconstruction network consistent, appropriate zero padding is used for all convolutional layers.

[0111] The relationship between the intensity information of different polarization directions in I HRmos and S0, DOLP, and AOP is

[0112]

[0113]

[0114]

[0115] Using the relationship between the polarization degree and S0, DoLP, and AOP, a non-linear mapping relationship between them can be established through a deep learning network.

[0116] Step 2: Input the reconstruction result of Step 1 and the polarization characteristic images of the real S0, DOLP, and AOP into the discriminator D1 in the second-layer generative adversarial network for evaluation.

[0117] AsFigure 9 As shown in Figure 9 , the discriminator D1 consists of five convolutional layers and one fully-connected layer. The first convolutional layer has 16 convolutional kernels of size 28×28, the activation function of the first layer is Leaky-ReLU, and the sampling stride is 2. The second convolutional layer has 32 convolutional kernels of size 7×7, the activation function of the second layer is Leaky-ReLU, and the sampling stride is 2. The third convolutional layer has 48 convolutional kernels of size 5×5, the activation function of the third layer is Leaky-ReLU, and the sampling stride is 1. The fourth convolutional layer has 64 convolutional kernels of size 3×3, the activation function of the fourth layer is Leaky-ReLU, and the sampling stride is 1. The fifth convolutional layer has 128 convolutional kernels of size 3×3, the sampling stride is 1, and the result of the fifth layer is used as the output of the discriminator D1, the activation function is the Sigmoid function, and the sampling stride is 1. The outputs of all convolutional layers are processed by a batch normalization layer and then input into the activation function. Appropriate zero-padding is used in all convolutional layers.

[0118] Step 3: Calculate the loss functions of the generator G1 and the discriminator D1, and perform alternating reconstruction training on the generator G1 and the discriminator D1 respectively with the Adam optimizer according to the calculated loss functions, while updating the parameters inside the generator G1 and the discriminator D1.

[0119] Among them, the adversarial loss function of the generator G1 is:

[0120]

[0121] The L1 smooth loss function of the generator G1 is:

[0122] L smooth-L1 =smooth L1 (H - G1(I HRmos )) (24)

[0123]

[0124] The pixel loss function of the generator G1 is:

[0125]

[0126] In the formula, the total loss function of the generator G1 is:

[0127]

[0128] The loss function of the discriminator D1 is:

[0129]

[0130] In the formula, d represents the gradient of the image, and H is the expected high-resolution non-mosaic polarized image.

[0131] Step 4: When the error of the training result reaches the set error, the iterative training of reconstruction is completed, the final reconstruction result is output, and the generator G1 and the discriminator D1 are saved.

[0132] Finally, the overall network outputs high-resolution S0, DOLP, and AOP images with a resolution of 1920×1080.

[0133] As Figure 10 、 11 shown, the original polarization image I captured by the polarization detector (mos，k) passes through the first layer of the generative adversarial network to obtain a high-resolution polarization mosaic image I HRmos , and the resolution of the high-resolution polarization mosaic image I HRmos is higher than that of the original polarization image I (mos，k) , and the high-resolution polarization mosaic image I HRmos has richer detail features. Figure 12 、 14 、16 are the S0, DOLP, and AOP images respectively. These images are obtained by using the traditional polarization demosaicing method from the original polarization image I (mos，k) to obtain intensity images in different polarization directions, and then the polarization characteristic images are calculated according to the formula. Figure 13 、 16 、17 are the S0, DOLP, and AOP images generated by the method of the present invention respectively. It can be seen that the quality of the polarization characteristic images generated by this method is much higher than that of the images obtained by the traditional polarization demosaicing method.

[0134] It should be understood that the above description is relatively detailed with the DMD size of 1920×1080, the resolution of the sub-focal plane polarization detector of 960×540, and the 2-fold super-resolution image reconstruction as examples, and it should not be considered as a limitation to the protection scope of the present invention patent. Those skilled in the field of polarization detection, under the inspiration of the present invention and without departing from the scope protected by the claims of the present invention, make substitutions or deformations, which all fall within the protection scope of the present invention. The scope of protection requested by the present invention shall be subject to the appended claims.

Claims

1. A compressive sensing polarization super-resolution imaging method based on a deep learning framework, characterized in that, The method comprises the following three steps: compressive coded sampling, super-resolution reconstruction, and depolarization demosaicing; among which, the super-resolution reconstruction and depolarization demosaicing stages are implemented by a two-layer generative adversarial network; Step 1, compressive encoding sampling: Use a digital micromirror device (DMD) array to perform compressive sensing encoding measurement sampling on the imaging scene. First, project the imaging scene onto the DMD, and perform independent block encoding on the imaging scene through the DMD. Each block area after segmentation is imaged on a pixel of the sub-focal plane polarization detector. The sub-focal plane polarization detector collects the encoded image information to generate the original polarization mosaic image I (mos，k) ; Step 2, super-resolution reconstruction: The polarization directions between adjacent pixels in the original polarization mosaic image are different. The pixels with polarization degrees of 0°, 45°, 90°, and 135° are segmented as a super-pixel, and then recombined into a one-dimensional measurement value vector y Ki ; and the corresponding coding mask is recombined into a measurement matrix Φ for reconstruction; the measurement vector y Ki and the measurement matrix Φ are used as input data and fed into the generator G0 of the first-layer generative adversarial network for compressive sensing reconstruction to generate a pseudo high-resolution image I HRmos , and the generated pseudo-image and the real high-resolution polarization image are discriminated for authenticity in the discriminator D0. The network parameters in G0 and D0 are iteratively updated according to the loss function, so that the generated image continuously approaches the real high-resolution polarization image; Step 3, depolarize the mosaic: Input the image I generated in Step 2, G0 HRmos into the generator G1 of the second layer of the generative adversarial network to establish the mapping relationship between the polarized mosaic image and the polarization characteristics, thereby generating the depolarized mosaic S0, DOLP, and AOP images. Evaluate the generated S0, DOLP, and AOP images and the real S0, DOLP, and AOP images in the discriminator D1, and iteratively update the network parameters in G1 and D1 according to the loss function to make the generated images continuously approach the real high-resolution S0, DOLP, and AOP images.

2. The compressive sensing polarization super-resolution imaging method based on a deep learning framework according to claim 1, characterized in that, The specific steps of the first step of compressive coded sampling are as follows: Step 1: The digital micromirror device (DMD) array encodes the imaging scene. First, the imaging scene is projected onto the DMD, and the DMD performs independent block encoding on the imaging scene. Each block after segmentation is imaged on a pixel of the sub-focal plane polarization detector, and a compressed-sampled polarization mosaic raw image I is obtained through the sub-focal plane polarization camera. (mos，k) , and the number of obtained polarization mosaic raw images is equal to the number of encoding measurements T. Let k represent the k-th measurement. Step 2: Obtain the training set data of the reconstructed image: 2.1, Original polarized mosaic image I (mos，k) The degree of polarization between adjacent pixels is different. The obtained original polarized mosaic image I (mos，k) is segmented with pixels having polarization degrees of 0°, 45°, 90°, and 135° as a super-pixel region. The value of the super-pixel is converted into a one-dimensional measurement value vector y i , y i represents the i-th super-pixel; at the same time, the corresponding coded mask matrix is reorganized to generate a coded matrix M for reconstruction i ; 2.

2. Combine the T superpixel measurement value vectors y at the same position on different polarization-encoded images i into a vector y Ki , and the encoding matrix M i to form the measurement matrix Φ, y Ki and Φ are used as the input data of the generator; 3. The compressive sensing polarization super-resolution imaging method based on a deep learning framework according to claim 1, characterized in that, The second step of super-resolution reconstruction specifically includes the following steps: Step 1: Input the measurement vector y Ki and the measurement matrix Φ into the generator G0 of the first-layer generative adversarial network. Perform a linear mapping on y Ki through a fully connected layer to obtain an initial reconstruction result After that reconstruct and generate an output result after passing through four convolutional layers Among them, the initial reconstruction of the image patch in the superpixel region using the measurement value y Ki and the measurement matrix Φ is realized by using a fully connected network; The expression is In the expression, represents the initial reconstruction result, W p represents the weight parameter matrix of the fully connected layer, b i represents the bias; The generator G0 is composed of a fully connected layer and four convolutional layers. The fully connected layer is used to perform preliminary reconstruction on the compressed sampling information; the first two convolutional layers extract features of the image, the third convolutional layer is a sub-pixel convolutional layer to enhance the resolution of the image, and the fourth convolutional layer outputs the reconstruction result; the activation functions of the first three convolutional layers are ReLU, and the activation function of the fourth layer is the hyperbolic tangent function tanh; Step 2: Input the reconstruction result of Step 1 and the real image patch into the discriminator D0 of the first-layer generative adversarial network for discrimination; the discriminator D0 has 4 convolutional layers. The activation functions of the first three convolutional layers are Leaky-ReLU; the result of the fourth layer is used as the output of the discriminator D0, and the activation function is the Sigmoid function; Step 3: Calculate the loss functions of the generator G0 and the discriminator D0, and use the calculated loss to alternately perform reconstruction training on the generator G0 and the discriminator D0 using the Adam optimizer, and simultaneously update the parameters inside the generator G0 and the discriminator D0; Among them, the adversarial loss of the generator G0 is: The content loss of the generator G0 is: In the formula, the total loss function of the generator G0 is: The loss function of the discriminator D0 is: In the formula, φ is the pre-trained VGG-16, is the low-resolution image, is the high-resolution image, and C, H, and W represent the number of channels, the image length, and the image width, respectively; Step 4: When the error of the training result reaches the set error, the reconstruction iterative training is completed, and the generator G0 and the discriminator D0 are saved; Step 5: Stitch the reconstructed image patches into a complete polarization imaging scene image I HRmos , at this time, I HRmos is a high-resolution polarization intensity image with a polarization mosaic.

4. The compressive sensing polarization super-resolution imaging method based on a deep learning framework according to claim 1, characterized in that, The third step of depolarization demosaicing specifically includes the following steps: Step 1: Input the high-resolution polarization mosaic intensity image I HRmos into the generator G1 in the second-layer generative adversarial network; I HRmos has different polarization directions on adjacent pixels, and each superpixel contains polarization degrees of 0°, 45°, 90°, and 135°; Using the relationship between the intensity information of different polarization directions in I HRmos and S0, DOLP, and AOP, establish the mapping relationship between the high-resolution polarization mosaic intensity image I HRmos and the high-resolution polarization characteristic image, and generate the polarization characteristic images I S0 、I DOLP 、I AOP ; Among them, G1 contains two convolutional layers and two residual modules. One convolutional layer is used for preliminary feature extraction, and one convolutional layer is used for result output; the activation function of the convolutional layer is ReLU, and the activation function of the output convolutional layer is the hyperbolic tangent function tanh; I HRmos The relationship between the intensity information in different polarization directions and S0, DOLP, and AOP is Step 2: Input the generation result of Step 1 and the polarization characteristic images of the real S0, DOLP, and AOP into the discriminator D1 in the second-layer generative adversarial network for discrimination; the discriminator D1 is composed of 5 convolutional layers, and the activation functions of the first four convolutional layers are Leaky-ReLU; the activation function of the fifth convolutional layer is the Sigmoid function; Step 3: Calculate the loss functions of the generator G1 and the discriminator D1, and use the calculated loss to alternately perform reconstruction training on the generator G1 and the discriminator D1 using the Adam optimizer function, and simultaneously update the parameters inside the generator G1 and the discriminator D1; Among them, the adversarial loss function of the generator G1 is: The L1 smooth loss function of the generator G1 is: L smooth-L1 = smooth L1 (H - G1(I HRmos ))(10) The pixel loss function of the generator G1 is: In the formula, the total loss function of the generator G1 is: The loss function of the discriminator D1 is: In the formula, d represents the gradient of the image, and H is the expected high-resolution demosaiced polarization image; Step 4: When the error of the training result reaches the set error, the reconstruction iterative training is completed, the final reconstruction result is output, and the generator G1 and the discriminator D1 are saved.

Images