A Deep Learning-Based Super-Resolution Reconstruction Method for Polarization Images

Abstract

This invention discloses a deep learning-based super-resolution reconstruction method for polarization images, which can be used to reconstruct corresponding high-resolution polarization images from low-resolution polarization images. The method includes: acquiring high-resolution polarization images using a polarization camera and preprocessing them to obtain low-resolution polarization images, creating a dataset; fusing feature information through a shallow convolutional neural network; performing dense feature fusion through a residual dense module; using an upscaling module to amplify the number of image feature channels to a corresponding magnification factor; obtaining the super-resolution polarization image through an output module; and optimizing the neural network using a polarization-aware and pixel-mixing loss function. This method, based on a deep learning convolutional neural network, constructs a super-resolution reconstruction neural network model and uses a polarization-aware and pixel-mixing loss function to train and optimize the neural network, enabling the reconstruction of super-resolution polarization images from low-resolution polarization images.

Description

Technical Field

[0001] This invention relates to the field of polarization imaging technology, and in particular to a deep learning-based method for super-resolution reconstruction of low-resolution polarization images. Background Technology

[0002] Polarization imaging technology can acquire polarization information in a target scene, thus solving the problem that target information is difficult to represent using light intensity and wavelength information due to environmental constraints. Currently, the low resolution of commonly used polarization imaging systems is mainly due to hardware limitations, or downsampling of the transmitted image to ensure transmission speed and save space. Super-resolution reconstruction of polarization images is of great significance for reconstructing high-resolution polarization images and obtaining more accurate polarization information, such as linear polarization degree and polarization angle.

[0003] Currently, super-resolution reconstruction methods for traditional unpolarized images are mainly divided into three categories: interpolation-based methods, reconstruction-based methods, and learning-based methods. Interpolation-based methods generate and reconstruct surrounding pixels based on the pixel information of the original image. Commonly used methods include nearest-neighbor interpolation, bilinear interpolation, and bicubic interpolation. Reconstruction-based methods utilize prior knowledge of the image to find a degradation model from a high-resolution polarized image to a low-resolution polarized image and apply the inverse process to the low-resolution polarized image to obtain a super-resolution polarized image. However, the above two types of methods cannot fully utilize the feature information in the image, so the results of super-resolution reconstruction are not ideal, especially for high magnification factors (such as 4x and above), where the generated image suffers from blurred details and textures, severe jagged edges, and significant noise.

[0004] Unpolarized image super-resolution reconstruction based on deep learning has become a research hotspot in the field of super-resolution. However, directly applying the current super-resolution reconstruction neural network to polarized image super-resolution reconstruction has problems such as failing to fully utilize polarization information, focusing too much on the recovery of light intensity information while ignoring the recovery of polarization information, resulting in unsatisfactory reconstructed polarized images. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art, solve the problem of difficulty in recovering polarization information in super-resolution reconstruction of low-resolution polarization images, and provide a deep learning-based super-resolution reconstruction method for polarization images.

[0006] The objective of this invention is achieved through the following technical solution:

[0007] A deep learning-based method for super-resolution reconstruction of polarization images includes the following steps:

[0008] S1. Use a polarization camera to capture a single-channel high-resolution polarization image A, preprocess the acquired polarization image A, and construct a dataset. Divide the dataset into a training set, a test set, and a validation set according to the proportions.

[0009] S2. Build a neural network model and set a loss function that combines polarization perception and pixel mixing, utilizing polarization feature information and polarization pixel information in the dataset;

[0010] S3. Input the dataset into the neural network model to train the network and obtain the super-resolution reconstruction neural network model;

[0011] S4. Test the test set using the super-resolution reconstruction neural network model, and obtain a high-resolution output polarization image with a pre-set magnification factor from the output layer of the super-resolution reconstruction neural network model.

[0012] Further, the preprocessing process in step S1 is as follows: First, each polarization image A is split into four sub-images according to the polarization direction and merged into a high-resolution polarization image B with four channels. By cropping, the length and width of polarization image B are divisible by the magnification factor. Then, bicubic interpolation downsampling is used to reduce polarization image B by a fixed factor to obtain a low-resolution polarization image C with four channels. Finally, polarization image C is cropped into a 32×32 image block, polarization image B with a magnification factor of 4 is cropped into a 128×128 image block, and polarization image B with a magnification factor of 3 is cropped into a 96×96 image block. The image blocks of polarization image B and polarization image C are used as image pairs to form a dataset.

[0013] Furthermore, the neural network model includes:

[0014] The shallow feature extraction module includes two convolutional layers. The first convolutional layer consists of 64 3×3 convolutional kernels, and the second convolutional layer also consists of 64 3×3 convolutional kernels.

[0015] There are 16 residual dense modules. Each residual dense module consists of a 3×3 convolutional layer, a linear correction unit ReLU function, a connection layer for feature fusion, and a 1×1 convolutional layer. The layers in each residual dense module are connected in a dense manner to combine features. Finally, the input before each residual dense module is added to the output after passing through the corresponding residual dense module to obtain the local residual connection result.

[0016] The global feature fusion module includes a connection layer, a 1×1 convolutional layer, and a 3×3 convolutional layer for feature fusion. The connection layer and the 1×1 convolutional layer fuse all features output from several residual dense modules, and the 3×3 convolutional layer further extracts features from the fusion result.

[0017] The global residual learning module adds the output of the first convolutional layer of the shallow feature extraction module to the output of the global feature fusion module, and uses short-circuit connections to achieve global residual learning.

[0018] The upscaling module, including 5×5 convolutional layers and 3×3 convolutional layers, can upscale the obtained coarse-resolution features to fine-resolution features, and amplify the number of channels of the feature information to a multiple of the corresponding amplification factor.

[0019] The output module includes subpixel convolutional layers and 3×3 convolutional layers. The subpixel convolutional layers amplify the length and width of the feature information to a multiple of the corresponding amplification factor. The final 3×3 convolutional layer is used to reconstruct a four-channel high-resolution polarization image, resulting in the final output super-resolution polarization image.

[0020] Furthermore, the steps for training the neural network model in step S3 are as follows:

[0021] First, the polarization image C is input into the neural network model. The shallow feature extraction module extracts and fuses features from the polarization images C in four polarization directions, and transmits the obtained feature information to the residual dense module, outputting feature information at different levels. The global feature fusion module fuses the output feature information at different levels and extracts features from the fused features. The global residual learning module adds the obtained feature information to the feature information output by the shallow feature extraction module to obtain image feature information in low-resolution space. These image feature information are then input into the upscaling module to increase the number of channels of the feature information to a set magnification factor, and the output module obtains the super-resolution reconstruction neural network model in high-resolution space.

[0022] Furthermore, the polarization sensing and pixel hybrid loss function includes total pixel loss, polarization pixel loss, and polarization sensing loss. The total pixel loss is used to recover light intensity information, the polarization pixel loss is used to recover polarization information, and the polarization sensing loss is used to improve the perception effect of polarization information.

[0023] Furthermore, the mean square error of the pixel values ​​of the output polarization image of the neural network model and the high-resolution polarization image of the four channels is used as the total pixel loss;

[0024] The mean square error of pixel values ​​in the adaptively adjusted linear polarization degree image and polarization angle image is used as the polarization pixel loss, where weights are assigned according to the magnitude of the mean square error of the linear polarization degree and polarization angle, and the weights of the two are dynamically adjusted.

[0025] The pre-trained VGG network is used to extract polarization image features. Three identical single-channel linear polarization images are superimposed to transform them into a three-channel polarization image, which is then input into the neural network model. The polarization features of the linear polarization image reconstructed from the output polarization image and the linear polarization image reconstructed from polarization image A are extracted. The mean square error of the two is used as the linear polarization degree perception loss. The same operation is performed on the single-channel polarization angle image to obtain the polarization angle perception loss. The linear polarization degree perception loss and the polarization angle perception loss are combined to obtain the polarization perception loss.

[0026] The total pixel loss, polarization pixel loss, and polarization sensing loss are combined according to empirically obtained weights to form the final polarization sensing and pixel hybrid loss function.

[0027] Furthermore, the parameters of the neural network model are adjusted to meet the requirements for the output polarization image resolution, including the initial learning rate, batch size, loss function constraints, number of residual dense modules, and number of convolutional kernels.

[0028] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the deep learning-based polarization image super-resolution reconstruction method.

[0029] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep learning-based polarization image super-resolution reconstruction method.

[0030] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0031] 1. This invention primarily employs a residual dense structure to build a neural network. The neural network learns polarization image features in low-resolution space, which reduces the space resources required for learning compared to learning polarization images in high-resolution space, thus improving the learning speed. The residual dense structure can connect features from multiple levels, thereby fully utilizing the polarization information of the low-resolution polarization image, making the output super-resolution polarization image closer to the high-resolution polarization image.

[0032] 2. This invention primarily uses the mean square error between the output image and the real image pixels as the overall pixel loss, the mean square error of the polarization information pixels whose weights are adaptively and dynamically adjusted during training as the polarization pixel loss function, and the mean square error of the polarization information features based on the pre-trained VGG network as the polarization perception loss. By incorporating a polarization perception and pixel hybrid loss function, the pixel and feature information of image polarization information can be effectively utilized, thereby achieving better polarization information perception in super-resolution reconstruction. Attached Figure Description

[0033] Figure 1 This is a schematic flowchart of the method of the present invention;

[0034] Figure 2 This is a schematic diagram of the structure of the neural network model in this embodiment;

[0035] Figures 3a to 3i This is a schematic diagram comparing the four-fold super-resolution reconstruction effect of a low-resolution polarization image according to an embodiment of the present invention; wherein... Figure 3a This is an intensity map obtained by upsampling a low-resolution polarization image using bicubic interpolation. Figure 3b For the corresponding linear polarization degree diagram, Figure 3c For the corresponding polarization angle diagram, Figure 3d , 3e 3f represents the light intensity map, linear polarization degree map, and polarization angle map obtained by the neural network model after super-resolution reconstruction in the usage example, respectively. Figure 3g , 3h 3i and 3i are the light intensity map, linear polarization map, and polarization angle map reconstructed from the label image, respectively, which are the true value images.

[0036] Figures 4a to 4c This is a schematic diagram of the magnified image of the intensity map obtained by four times super-resolution reconstruction of a low-resolution polarization image according to an embodiment of the present invention. Figure 4a A magnified view of a local area of ​​the light intensity map for the target label; Figure 4b This is a magnified comparison diagram of region A in the light intensity map; Figure 4c This is a magnified comparison diagram of region B in the light intensity map.

[0037] Figures 5a to 5c This is a schematic diagram of the magnified image of the linear polarization map obtained by four times super-resolution reconstruction of a low-resolution polarization image according to an embodiment of the present invention. Figure 5a A magnified view of a local area of ​​the linear polarization degree map of the target label;

[0038] Figure 5b This is a magnified comparison diagram of region A in the linear polarization degree map. Figure 5c This is a magnified comparison diagram of region B in the linear polarization degree map.

[0039] Figures 6a to 6c This is a schematic diagram showing a local magnified effect of the polarization angle map obtained by four times super-resolution reconstruction of a low-resolution polarization image according to an embodiment of the present invention. Figure 6a This is a magnified view of a local area of ​​the polarization angle diagram of the target label. Figure 6b This is a magnified comparison diagram of region A in the polarization angle diagram; Figure 6c This is a magnified comparison diagram of region B in the polarization angle diagram. Detailed Implementation

[0040] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0041] The following describes, with reference to the accompanying drawings, a deep learning-based polarization image super-resolution reconstruction method according to an embodiment of the present invention, taking focal plane polarization image super-resolution reconstruction as an example.

[0042] Figure 1 This is a flowchart of a deep learning-based polarization image super-resolution reconstruction method provided by the present invention. The image super-resolution reconstruction method in this embodiment includes:

[0043] Step 1: Construct a polarization image dataset: High-resolution polarization images are captured using a split-plane polarization camera. Data preprocessing is then performed. Each polarization image A is first split into four sub-images according to the polarization direction and merged into a high-resolution polarization image B with four channels. The image B is then cropped so that its length and width are divisible by the magnification factor. Next, bicubic interpolation downsampling is used to reduce the polarization image B by a fixed factor, resulting in a four-channel low-resolution polarization image C. Finally, polarization image C is cropped into 32×32 image blocks, polarization image B with a 4x magnification factor is cropped into 128×128 image blocks, and polarization image B with a 3x magnification factor is cropped into 96×96 image blocks. Image blocks from polarization image B and polarization image C are used as image pairs to form the dataset.

[0044] Assuming that in each channel of the input image, the high-resolution polarization image B of each channel is represented as... The low-resolution polarization image C for each channel is represented as follows: Both conditions are met:

[0045]

[0046] Where θ∈{0°, 45°, 90°, 135°}, k θ This represents the degradation model matrix, which in this embodiment is the weight matrix of bicubic interpolation. Finally, the dataset is split into a training set, a validation set, and a test set in an 8:1:1 ratio.

[0047] Step 2: Build a neural network model, such as Figure 2As shown, the neural network model mainly consists of multiple residual dense modules, and is constrained by polarization sensing and pixel mixing loss functions. The neural network model is optimized and trained to obtain a super-resolution reconstruction neural network model.

[0048] Step 201: Configure the shallow feature extraction module. Specifically, the shallow feature extraction module includes two convolutional layers. The first convolutional layer consists of 64 3×3 convolutional kernels, and the second convolutional layer also consists of 64 3×3 convolutional kernels.

[0049] Step 202: Configure residual dense modules. Specifically, in this embodiment, a total of 16 residual dense modules are configured. Each residual dense module consists of a 3×3 convolutional layer, a ReLU function, a connection layer for feature fusion, and a 1×1 convolutional layer. Features are combined between layers in each residual dense module using dense connections. Finally, the input before each residual dense module is added to the output after passing through that module to obtain the local residual connection result. In this embodiment, each residual dense module contains 8 densely connected convolutional layers and their corresponding ReLU functions.

[0050] Step 203: Configure the global feature fusion module. Specifically, the global feature fusion module includes a connection layer, a 1×1 convolutional layer, and a 3×3 convolutional layer for feature fusion. The connection layer and the 1×1 convolutional layer for feature fusion fuse all features output from multiple residual dense modules, and the 3×3 convolutional layer further extracts features from the fusion result.

[0051] Step 204: Configure the global residual learning module. The global residual learning module adds the output of the first shallow feature extraction layer to the output of the global feature fusion module, and uses short-circuit connections to achieve global residual learning.

[0052] Step 205: Configure the upscaling module. This includes 5×5 convolutional layers and 3×3 convolutional layers, which can upscale the obtained coarse-resolution features to fine-resolution features, increasing the number of channels of the feature information to a multiple of the corresponding magnification factor.

[0053] Step 206: Configure the output module. Specifically, the output module consists of sub-pixel convolutional layers and 3×3 convolutional layers. The sub-pixel convolutional layers amplify the length and width of the feature information to a multiple of the corresponding amplification factor. The final 3×3 convolutional layer is used to reconstruct a four-channel high-resolution polarization image, resulting in the final output super-resolution polarization image.

[0054] Step 3: Design a polarization sensing and pixel hybrid loss function. The light intensity information is recovered through the overall pixel loss, the polarization information is recovered through the polarization pixel loss, and the polarization sensing loss is used to improve the perception effect of polarization information.

[0055] Specifically, the overall pixel loss function The definition is as follows:

[0056]

[0057] Where N is the number of training samples, C is the number of image channels, and m and n are the image dimensions, respectively. Let θ represent the training pair of low-resolution polarization image C and high-resolution polarization image B, where θ∈{0°, 45°, 90°, 135°}, Θ represents the hyperparameters of the neural network model, and F function represents the super-resolution mapping obtained from the training pair.

[0058] Furthermore, in this embodiment of the invention, Stokes vectors are utilized. Characterizing polarization information, With polarization images I0, I in four polarization directions 45 I 90 I 135 The relationship is as follows:

[0059]

[0060] The polarization information, including light intensity S0, linear polarization degree DoLP, and polarization angle AoP, is represented by the Stokes vector. The light intensity S0 can be directly obtained by formula (3), and the linear polarization degree DoLP and polarization angle AoP are expressed by the following formulas:

[0061]

[0062]

[0063] Polarization pixel loss Pixel loss due to linear polarization and polarization angle pixel loss Composition, defined as follows:

[0064]

[0065]

[0066] Among them, DoLP pred and AoP pred The linear polarization degree image and polarization angle image are calculated from the super-resolution polarization image predicted by the neural network model (the output polarization image of step S4), respectively. DoLP gt and AoP gt The linear polarization degree image and polarization angle image are calculated from the actual high-resolution polarization image, respectively. μ is the polarization angle adjustment parameter, so that the pixel loss of linear polarization degree and the pixel loss of polarization angle are on the same order of magnitude in the initial training values.

[0067] Polarization pixel loss for:

[0068]

[0069] It should be noted that the weight w DoLP w AoP As the value of the corresponding loss function changes, the contribution of each sub-loss to the overall loss is evaluated through adaptive weighted loss. Sub-losses that contribute more are given higher weights, thereby balancing the numerical values ​​of each sub-loss.

[0070] Furthermore, polarization sensing loss Loss perceived by linear polarization degree and polarization angle sensing loss Composition, defined as follows:

[0071]

[0072]

[0073] Here, feat(·) represents the polarization feature-aware mapping, which consists of a portion of the pre-trained VGG network layers. It is used to extract the polarization features of the input image to obtain a feature map, where m and n are the image dimensions, respectively. In this embodiment, the portion of the pre-trained VGG-19 network before the first pooling layer is used.

[0074] Furthermore, polarization sensing loss for:

[0075]

[0076] Furthermore, the polarization sensing and pixel fusion loss function L is:

[0077]

[0078] Where λ and η are coefficients that balance different loss terms.

[0079] Step 4: In this embodiment of the invention, the model parameters are adjusted according to the requirements of super-resolution reconstruction, including the initial learning rate, batch size, loss function constraints, number of residual dense modules, and number of convolutional kernels, etc., to train the neural network model. During training, the input image is flipped and rotated to augment the data. Specifically, in this embodiment, the minimum batch size is 16, the learning rate is initialized to 0.0001, the exponential decay learning rate method is used, the decay rate is set to 0.5, the training period is 105, the learning rate decays every 20 rounds, and the Adam algorithm is used to optimize the loss function.

[0080] In this embodiment of the invention, Peak Signal-to-Noise Ratio (PSNR) and Learning Perceptual Image Patch Similarity (LPIPS) are used as evaluation metrics. The higher the value of the metric (↑), the closer the image pixel level is to the label image; the lower the value (↓), the closer the image perception effect is to the label image. Ten four-channel polarization images with pixel dimensions of 306×256×4 and 408×341×4 are used as low-resolution polarization image inputs for neural networks with magnification factors of 4 and 3, respectively. A high-resolution polarization image with pixel dimensions of 1224×1024×4 is used as the target label image. The evaluation metrics are calculated using the reconstructed high-resolution polarization image and the target label's high-resolution polarization image. The reconstruction results with magnification factors of 4 and 3 are shown in the table below.

[0081]

[0082] Experimental results show that, compared with the traditional bicubic interpolation method, the peak signal-to-noise ratio of the reconstructed polarization information after super-resolution reconstruction of low-resolution polarization images using the present invention is significantly improved. Furthermore, the similarity of the learned perceptual image patches decreases, indicating better visual perception quality of the image.

[0083] In an embodiment of the present invention, after performing four-fold super-resolution reconstruction using a polarization image super-resolution reconstruction neural network, the resulting visualizations of the light intensity map, linear polarization degree map, and polarization angle map calculated based on polarization information are shown below. Figures 3a to 3i As shown, where Figure 3a This is an intensity map obtained by upsampling a low-resolution polarization image using bicubic interpolation. Figure 3b For the corresponding linear polarization degree diagram, Figure 3c For the corresponding polarization angle diagram, Figure 3d e and f are the light intensity map, linear polarization degree map, and polarization angle map obtained after super-resolution reconstruction using the embodiments of the present invention, respectively. Figure 3g h and i are the light intensity map, linear polarization map, and polarization angle map reconstructed from the label image, respectively, which are the true value images. Figures 4a to 4c , Figures 5a to 5c , Figures 6a to 6c The images show a comparison of magnified local effects for intensity maps, linear polarization degree maps, and polarization angle maps. As can be seen from the figures, the deep learning-based polarization image super-resolution reconstruction method proposed in this invention produces images with continuous edges and can recover image details in polarization information that are difficult to recover using traditional interpolation methods. This method can effectively improve the resolution of low-resolution polarization images, and for polarization degree and polarization angle images with severely damaged details, it can recover their polarization information and improve their perceptual effect.

[0084] Preferably, embodiments of this application also provide a specific implementation of an electronic device capable of implementing all steps in the deep learning-based polarization image super-resolution reconstruction method described in the above embodiments. The electronic device specifically includes the following components:

[0085] Processor, memory, communications interface, and bus;

[0086] The processor, memory, and communication interface communicate with each other via a bus; the communication interface is used to realize information transmission between server-side devices, metering devices, and user-side devices.

[0087] The processor is used to call the computer program in memory. When the processor executes the computer program, it implements all the steps in the deep learning-based polarization image super-resolution reconstruction method in the above embodiments.

[0088] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the deep learning-based polarization image super-resolution reconstruction method in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the deep learning-based polarization image super-resolution reconstruction method in the above embodiments.

[0089] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0090] While this application provides method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive labor. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual device or client product execution, the method can be executed sequentially as shown in the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment).

[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0095] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention. Without departing from the spirit and scope of the claims, those skilled in the art can also make many specific modifications based on the teachings of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A deep learning-based method for super-resolution reconstruction of polarization images, characterized in that, Includes the following steps: S1. A single-channel high-resolution polarization image A is captured using a polarization camera. The acquired polarization image A is preprocessed, and a dataset is constructed. The dataset is divided into training, testing, and validation sets according to the proportions. The preprocessing process is as follows: First, each polarization image A is split into four sub-images according to the polarization direction and merged into a high-resolution polarization image B with four channels. By cropping, the length and width of polarization image B are divisible by the magnification factor. Then, bicubic interpolation downsampling is used to reduce polarization image B by a fixed factor to obtain a four-channel low-resolution polarization image C. Finally, polarization image C is cropped into a 32×32 image block, polarization image B with a magnification factor of 4 is cropped into a 128×128 image block, and polarization image B with a magnification factor of 3 is cropped into a 96×96 image block. The image blocks of polarization image B and polarization image C are used as image pairs to form the dataset. S2. Build a neural network model and set a polarization sensing and pixel mixing loss function, utilizing polarization feature information and polarization pixel information in the dataset; the polarization sensing and pixel mixing loss function includes total pixel loss, polarization pixel loss and polarization sensing loss, the total pixel loss is used to recover light intensity information, the polarization pixel loss is used to recover polarization information, and the polarization sensing loss is used to improve the perception effect of polarization information. S3. Input the dataset into the neural network model to train the network and obtain the super-resolution reconstruction neural network model; use the mean square error of the pixel values ​​of the output polarization image of the neural network model and the high-resolution polarization image of the four channels as the overall pixel loss; The mean square error of pixel values ​​in the adaptively adjusted linear polarization degree image and polarization angle image is used as the polarization pixel loss, where weights are assigned according to the magnitude of the mean square error of the linear polarization degree and polarization angle, and the weights of the two are dynamically adjusted. The pre-trained VGG network is used to extract polarization image features. Three identical single-channel linear polarization images are superimposed to transform them into a three-channel polarization image, which is then input into the neural network model. The polarization features of the linear polarization image reconstructed from the output polarization image and the linear polarization image reconstructed from polarization image A are extracted. The mean square error of the two is used as the linear polarization degree perception loss. The same operation is performed on the single-channel polarization angle image to obtain the polarization angle perception loss. The linear polarization degree perception loss and the polarization angle perception loss are combined to obtain the polarization perception loss. The total pixel loss, polarization pixel loss, and polarization sensing loss are combined according to empirically obtained weights to form the final polarization sensing and pixel fusion loss function. S4. Test the test set using the super-resolution reconstruction neural network model, and obtain the output polarization image with a pre-set magnification factor from the output layer of the super-resolution reconstruction neural network model.

2. The deep learning-based polarization image super-resolution reconstruction method according to claim 1, characterized in that, The neural network model includes: The shallow feature extraction module includes two convolutional layers. The first convolutional layer consists of 64 3×3 convolutional kernels, and the second convolutional layer also consists of 64 3×3 convolutional kernels. There are 16 residual dense modules. Each residual dense module consists of a 3×3 convolutional layer, a linear correction unit ReLU function, a connection layer for feature fusion, and a 1×1 convolutional layer. The layers in each residual dense module are connected in a dense manner to combine features. Finally, the input before each residual dense module is added to the output after passing through the corresponding residual dense module to obtain the local residual connection result. The global feature fusion module includes a connection layer, a 1×1 convolutional layer, and a 3×3 convolutional layer for feature fusion. The connection layer and the 1×1 convolutional layer fuse all features output from several residual dense modules, and the 3×3 convolutional layer further extracts features from the fusion result. The global residual learning module adds the output of the first convolutional layer of the shallow feature extraction module to the output of the global feature fusion module, and uses short-circuit connections to achieve global residual learning. The upscaling module, including 5×5 convolutional layers and 3×3 convolutional layers, can upscale the obtained coarse-resolution features to fine-resolution features, and amplify the number of channels of the feature information to a multiple of the corresponding amplification factor. The output module includes subpixel convolutional layers and 3×3 convolutional layers. The subpixel convolutional layers amplify the length and width of the feature information to a multiple of the corresponding amplification factor. The final 3×3 convolutional layer is used to reconstruct a four-channel high-resolution polarization image, resulting in the final output super-resolution polarization image.

3. The deep learning-based polarization image super-resolution reconstruction method according to claim 1, characterized in that, The steps for training the neural network model in step S3 are as follows: First, the polarization image C is input into the neural network model. The shallow feature extraction module extracts and fuses features from the polarization images C in four polarization directions, and transmits the obtained feature information to the residual dense module, outputting feature information at different levels. The global feature fusion module fuses the output feature information at different levels and extracts features from the fused features. The global residual learning module adds the obtained feature information to the feature information output by the shallow feature extraction module to obtain image feature information in low-resolution space. These image feature information are then input into the upscaling module to increase the number of channels of the feature information to a set magnification factor, and the output module obtains the super-resolution reconstruction neural network model in high-resolution space.

4. The deep learning-based polarization image super-resolution reconstruction method according to claim 1, characterized in that, The parameters of the neural network model are adjusted to meet the requirements for the output polarization image resolution, including the initial learning rate, batch size, loss function constraints, number of residual dense modules, and number of convolutional kernels.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the deep learning-based polarization image super-resolution reconstruction method according to any one of claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the deep learning-based polarization image super-resolution reconstruction method according to any one of claims 1 to 4.

Images