A spatial modulation type polarization image super-resolution reconstruction method based on heterogeneous feature fusion

Abstract

The application discloses a spatial modulation type polarized image super-resolution reconstruction method based on heterogeneous feature fusion, which comprises the following steps: 1, acquiring an interference intensity image with target scene polarization information, constructing a four-channel high-resolution true value image and a corresponding four-channel low-resolution input image; 2, constructing a spatial modulation type polarized image super-resolution reconstruction network based on heterogeneous feature fusion; 3, establishing a polarized image super-resolution total loss function; 4, taking the four-channel low-resolution heterogeneous polarized feature fusion image as input, training the network, and using the trained model to perform super-resolution reconstruction processing on the polarized image to be processed, so as to obtain a high-resolution polarized image. The application can realize the spatial modulation type polarized image super-resolution reconstruction based on heterogeneous feature fusion, effectively improve the reconstruction quality and detail fidelity of the polarized image, and thus can provide high-quality and clear polarized image data for polarized remote sensing, target detection and other advanced visual tasks.

Description

Technical Field

[0001] This invention belongs to the field of polarization imaging and image super-resolution technology, specifically a spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion. Background Technology

[0002] Polarization imaging systems measure the polarization state of light reflected or transmitted from a target object to obtain its polarization parameters, such as Stokes parameters (S0, S1, S2, S3), degree of polarization, and polarization angle. This allows them to reflect the object's boundary, surface features, shape, and material information. Polarization imaging systems have been widely used in remote sensing imaging, surface detection, image dehazing, underwater image restoration, and target recognition for autonomous driving.

[0003] Image super-resolution reconstruction aims to recover high-resolution images from low-resolution images. Compared to ordinary intensity images, polarization images contain additional polarization dimension information, providing richer features and constraints for image reconstruction. However, most existing super-resolution methods are designed for intensity images and do not fully consider the inherent physical constraints between polarization information, making them difficult to directly apply to polarization image super-resolution tasks. Furthermore, most existing polarization image super-resolution methods are applicable to focal plane polarization images, lacking specific methods for super-resolution reconstruction of other types of polarization imaging data, such as spatially modulated images. On the other hand, due to hardware design limitations, polarization cameras typically have lower resolution than ordinary RGB cameras, reducing the fidelity of imaging information.

[0004] Traditional methods for enhancing optical resolution primarily involve optimizing the optical imaging system or employing image super-resolution techniques for post-processing acquired images. However, improving imaging system resolution from a hardware improvement perspective is often limited by factors such as detector operating conditions and manufacturing processes. Adopting traditional optical imaging system design approaches for resolution enhancement typically requires a significant increase in hardware costs and may even hinder engineering applications. Furthermore, the size, pixel size, and response sensitivity of current photodetectors are approaching physical limits, leaving limited room for improving resolution by reducing pixel size or increasing the number of array elements. Without increasing system structural complexity, relying solely on optical optimization is insufficient to further improve imaging resolution.

[0005] With the goal of improving the performance of image super-resolution algorithms, this paper considers the imaging quality of the front-end optical system and its corresponding processing methods, and designs a back-end image super-resolution reconstruction algorithm. This algorithm is a collaborative effort between the front-end optical imaging and the back-end image processing algorithm of the imaging system. It can obtain better image reconstruction results, realize functions that are difficult to achieve by traditional imaging methods, such as imaging or measurement of polarization information, and improve performance indicators such as spatial resolution. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion. This method aims to achieve accurate reconstruction of high-resolution polarization images while adapting to the output of a front-end spatial modulation polarization imaging optical system. This will meet the image requirements of advanced visual tasks such as polarization target detection and scene analysis, providing high-fidelity polarization image data support and enhancing the practical application capabilities of polarization imaging technology in complex scenarios.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: The present invention provides a spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion, characterized by the following steps: Step 1: Obtain the interferometric intensity image with polarization information of the target scene, and construct a four-channel high-resolution ground truth image. and the corresponding four-channel low-resolution input image ; Step 2: Construct a back-end polarization image super-resolution network, including: a shallow feature extraction module (SFE), a deep feature extraction module (DFE), a global feature fusion module (GFF), and a high-resolution image reconstruction module (HRR), and then... The image was processed to obtain a super-resolution four-channel polarization image. ; Step 3, based on and Establish the total loss function of the back-end polarization image super-resolution reconstruction network. ; Step 4: Train the polarization image super-resolution reconstruction network using the ADAM optimizer and calculate the total loss function. The network parameters are updated, and training stops when the number of training iterations reaches a set number, thus obtaining a trained polarization image super-resolution reconstruction model, which is used for super-resolution reconstruction of low-resolution polarization images.

[0008] The present invention provides a method for super-resolution reconstruction of spatially modulated polarization images based on heterogeneous feature fusion, wherein step 1 is performed according to the following steps: Step 1.1: Demodulate the interference intensity image to obtain the Stokes parameter, which is used to calculate the polarization degree image and polarization angle image of the target scene; Step 1.2: Perform intensity mapping processing on the interference intensity image to obtain the original visual intensity image; Step 1.3: After normalizing the polarization degree image and polarization angle image respectively, we obtain the normalized polarization degree image and the normalized polarization angle image. Step 1.4: Perform feature fusion along the channel dimension on the four heterogeneous feature images: the original visual intensity image, the Stokes parameter total intensity image, the normalized polarization degree image, and the normalized polarization angle image, to construct a four-channel high-resolution ground truth image. ; Step 1.5: Downsample the original visual intensity image, the total intensity image of the Stokes parameter, the polarization degree image, and the polarization angle image respectively to obtain the low-resolution original visual intensity image R, the low-resolution total intensity image S0, the low-resolution polarization degree image, and the low-resolution polarization angle image. Step 1.6: After normalizing the low-resolution polarization degree image and the low-resolution polarization angle image respectively, we obtain the normalized low-resolution polarization degree image D and the normalized low-resolution polarization angle image A. Step 1.7: Perform feature fusion along the channel dimension for R, S0, D, and A to construct a four-channel low-resolution input image. .

[0009] Furthermore, step 2 is performed as follows: Step 2.1, Shallow Feature Extraction (SFE) module... Perform a convolution operation to obtain the initial features. Then, on Perform a convolution operation to obtain shallow features. ; Step 2.2: The deep feature extraction module (DFE) consists of G residual dense blocks, where the g-th residual dense block includes m convolutional units, a concatenation layer, and a convolutional layer with a 1×1 kernel; where g∈[1,G], each convolutional unit consists of a convolutional layer with a 3×3 kernel and a LeakyReLU activation function layer. When g=1, As the first Pre-sequence feature maps The input is placed into the g-th residual dense block and processed sequentially through m convolutional units to obtain m convolutional features; The m convolutional features are then concatenated along the channel dimension in a concatenation layer to obtain the g-th concatenated feature. This concatenated feature is then processed in a convolutional layer with a 1×1 kernel to obtain the g-th fused feature. Thus and Element-wise addition fusion is performed using residual joins to obtain the first... A dense feature map ; thus, the dense feature map output from the G residual dense blocks { After concatenating |g∈[1,G]} along the channel dimension, deep features are obtained. ; Step 2.3: The Global Feature Fusion (GFF) module consists of a concatenation layer, a 1×1 convolutional layer, and a 3×3 convolutional layer, which are then sequentially applied to... After processing, the obtained global residual features are then compared with... After performing residual connections, global fusion features are obtained. ; Step 2.4: The High-Resolution Image Reconstruction (HRR) module consists of the Feature Refinement Module (FRB) and the Upsampling Module (UPS), and performs... After processing, a super-resolution four-channel polarization image is obtained. .

[0010] Furthermore, step 2.4 is performed as follows: Step 2.4.1 The input to the Feature Refinement (FRB) module first passes through a 5×5 convolutional layer and a Leaky ReLU activation function layer, then through a 3×3 convolutional layer and a Leaky ReLU activation function layer, to obtain the refined features. ; Step 2.4.2: Refine the features The image is input into the upsampling module UPS, and then processed sequentially through r subpixel convolutional units and a convolutional layer with a 3×3 kernel, outputting a super-resolution four-channel polarization image. , where r∈[1,R], and each subpixel convolutional unit consists of a convolutional layer with a kernel of 3×3 and a subpixel convolutional layer.

[0011] Furthermore, step 3 is performed as follows: Step 3.1: Construct content loss using equation (1) : (1) In equation (1), represent Norm operations; Step 3.2: Construct polarization sensing loss using equation (2) : (2) In equation (2), and These represent the extraction operators for the degree of polarization and the polarization angle, respectively. This represents the feature map extracted from the last convolutional layer before the first max pooling layer in the VGG-19 network. Step 3.3: Construct the total loss function using equation (3). : (3) In equation (3), The parameters to be trained for the polarization image super-resolution network are . These are the weighting coefficients.

[0012] The present invention provides an electronic device, including a memory and a processor, characterized in that the memory is used to store a program supporting the processor in performing the method described therein, and the processor is configured to execute the program stored in the memory.

[0013] The present invention discloses a computer-readable storage medium storing a computer program, characterized in that the computer program is executed by a processor to perform the steps of the method described thereon.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a spatial modulation polarization image super-resolution deep neural network based on heterogeneous feature fusion, which is adapted to the output of the front-end spatial modulation polarization imaging optical system. This solves the problem that existing polarization image super-resolution methods are mostly only applicable to focal plane polarization images, thereby achieving high-resolution reconstruction of spatial modulation polarization images. 2. The spatial modulation polarization image super-resolution deep neural network based on heterogeneous feature fusion constructed in this invention fully utilizes the feature attributes of the original visual intensity image and polarization parameter images (total intensity image, degree of polarization image, and polarization angle image). By fusion of heterogeneous features, low-resolution polarization images are stitched together in the channel dimension to construct a four-channel heterogeneous polarization feature fusion input image, which enriches the network input features. This enables the network to learn both the original image detail information and polarization structured features simultaneously, thereby effectively enhancing the network's ability to represent multidimensional polarization features, suppressing the loss of polarization details and reconstruction distortion, and significantly improving the overall accuracy and fidelity of polarization image super-resolution reconstruction.

[0015] 3. The spatial modulation polarization image super-resolution deep neural network based on heterogeneous feature fusion constructed in this invention differs from intensity image-based super-resolution methods that only calculate a single loss between the output and the label. Instead, it uses content loss and polarization perception loss to jointly constitute the total loss function, thereby effectively matching polarization information images with real data and achieving synchronous and accurate reconstruction of polarization information and overall image quality. Attached Figure Description

[0016] Figure 1 This is a flowchart of the polarization image super-resolution reconstruction method of the present invention; Figure 2 This is a flowchart of the heterogeneous polarization feature construction of the present invention; Figure 3This is a schematic diagram of the overall imaging scheme of the spatial modulation polarization super-resolution computational imaging system of the present invention; Figure 4 This is a schematic diagram of the spatial modulation module of the present invention; Figure 5 This is a schematic diagram of the modules of the polarization image super-resolution reconstruction network of the present invention; Figure 6 This is a schematic diagram of the polarization image super-resolution reconstruction network structure of the present invention; Figure 7 This is a schematic diagram of the structure of the dense residual block of the present invention. Detailed Implementation

[0017] In this embodiment, a spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion aims to address the problems that existing intensity image-based super-resolution methods are difficult to directly apply to polarization images and that existing polarization image super-resolution methods are mostly only applicable to focal plane polarization images. By constructing a spatial modulation polarization image super-resolution reconstruction network based on heterogeneous feature fusion, an image super-resolution reconstruction model that can adapt to the output of a front-end spatial modulation polarization imaging optical system and effectively super-resolution polarization images is obtained. This enables high-precision super-resolution reconstruction of low-resolution polarization images, meeting the image requirements of advanced vision tasks. Specifically, for example... Figure 1 As shown, the method is performed according to the following steps: Step 1: Obtain the interferometric intensity image with polarization information of the target scene, and construct a four-channel high-resolution ground truth image. and the corresponding four-channel low-resolution input image ; and The process of constructing heterogeneous polarization features is as follows: Figure 2 As shown, it specifically includes: Step 1.1: Demodulate the interference intensity image to obtain the Stokes parameter, which is used to calculate the polarization degree image and polarization angle image of the target scene; Step 1.1.1: Use a spatial modulation module to encode the polarization information of the target scene to generate an interference intensity image; like Figure 3 As shown, this embodiment combines a front-end spatial modulation polarization imaging optical system with a back-end polarization image super-resolution reconstruction algorithm to obtain a high-resolution polarization image. The front-end includes an optical lens, a spatial modulation module, and an imaging sensor, which can acquire an interference intensity image containing all polarization information of the target scene in a single exposure.

[0018] Specifically, under natural lighting conditions, the light signals reflected or scattered by the target scene are converged by an optical lens and then enter the spatial modulation module. The spatial modulation module is used to encode the original polarization information in the light field into the interference intensity distribution, thereby encoding the Stokes parameters into a single-frame interference modulation image, which contains polarization interference fringe information and light intensity modulation information.

[0019] The spatial modulation module is a key component of the system, consisting of a first Savart polarizer (SP1), a half-wave plate (HWP), a second Savart polarizer (SP2), and an analyzer. Figure 4 As shown.

[0020] The incident beam is split into two linearly polarized beams by the first Savart polarizer, and then passes through a half-wave plate and a second Savart polarizer to obtain four linearly polarized beams. After passing through the analyzer, the four beams vibrate in the same direction, have the same frequency, and have a constant phase difference, thus satisfying the interference condition. Subsequently, an interference image is formed at the focal plane by the imaging lens.

[0021] Step 1.1.2: Record the interference intensity image using a CCD area array image sensor; After the interference intensity image is formed, it is acquired using a CCD area array image sensor. The CCD area array image sensor can perform polarization detection in different bands, including the visible-near-infrared band and the short-wave infrared band, thereby acquiring the interference intensity image for the corresponding band. In this embodiment, the imaging resolution for the visible-near-infrared band is 2048×2048, with a pixel size of 12 μm; the imaging resolution for the short-wave infrared band is 640×512, with a pixel size of 20 μm. To ensure imaging quality, the imaging system and the target scene are kept relatively stationary during image acquisition. The acquired image data is in 16-bit raw data format.

[0022] Step 1.1.3: Use digital signal processing algorithms to perform polarization demodulation on the interference intensity image to obtain the Stokes parameters; After obtaining the interference intensity image, using the theoretical knowledge of wave optics, the spatial modulation analytical expression of the interference intensity image can be derived as shown in equations (1)-(3): (1) (2) (3) In equations (1)-(3), The spatial position coordinates of the image plane, For space carrier frequency, for The conjugate of complex numbers, The lateral offset introduced for SP1 and SP2 The center wavelength of the incident light This is the back focal length of the imaging lens.

[0023] Based on the interferometric intensity image, polarization demodulation is performed using digital signal processing algorithms to obtain the Stokes parameters, including the following steps: (1) Two-dimensional Fourier transform: A two-dimensional discrete Fourier transform is performed on the interference intensity image to map the interference fringe information in the spatial domain to the frequency domain, resulting in the spectral expression shown in equation (4): (4) In equation (4), , , , They represent , , , Fourier transform, , For frequency domain coordinates, the frequency domain translation amount , The size of a single pixel in a CCD. The number of pixels contained in a CCD row (column) (by default, the number of pixels in a CCD row and column is the same).

[0024] (2) Frequency domain filtering: Demodulation is performed using spatial frequency filtering. A suitable two-dimensional Gaussian low-pass filter is set up to perform frequency filtering at different frequency points, and the corresponding frequencies are extracted respectively. , Spectral information.

[0025] (3) Inverse Fourier Transform: Performing a two-dimensional inverse Fourier transform on the filtered spectrum according to the formula reconstructs the Stokes parameters, where the complex form is obtained. The components are recovered by taking their real and imaginary parts respectively. : (5) In equation (5), Indicates taking the real part, This indicates taking the imaginary part.

[0026] Through the above processing, accurate demodulation of Stokes parameters from the interference intensity image is achieved; the value range of the demodulated Stokes parameters is [0, 255]. Step 1.1.4: Calculate the degree of polarization and polarization angle of the target scene based on the Stokes parameters; Based on the Stokes parameters S0~S3, the degree of polarization and polarization angle of the target scene can be calculated using equations (6) and (7) respectively: (6) (7) This yields the corresponding polarization degree image and polarization angle image; the polarization degree image has a value range of [0,1], and the polarization angle image has a value range of [0,π].

[0027] Step 1.2: Perform intensity mapping processing on the interference intensity image to obtain the original visual intensity image; in this embodiment, the original visual intensity image is an 8-bit image with a value range of [0, 255]. Step 1.3: After normalizing the polarization degree image and polarization angle image using equation (8), the normalized polarization degree image and normalized polarization angle image are obtained; the normalized polarization degree image and polarization angle image are both 8-bit images with a value range of [0, 255]: (8) Step 1.4: Stitch together the original visual intensity image, the Stokes parameter total intensity image, the normalized polarization degree image, and the normalized polarization angle image along the channel dimension, and perform feature fusion to construct a four-channel high-resolution ground truth image. .

[0028] Step 1.5: Downsample the original visual intensity image, the total intensity image of the Stokes parameter, the polarization degree image, and the polarization angle image respectively to obtain the low-resolution original visual intensity image R, the low-resolution total intensity image S0, the low-resolution polarization degree image, and the low-resolution polarization angle image.

[0029] Step 1.6: After normalizing the low-resolution polarization degree image and the low-resolution polarization angle image respectively, we obtain the normalized low-resolution polarization degree image D and the normalized low-resolution polarization angle image A. Step 1.7: Concatenate four heterogeneous feature images (R, S0, D, and A) along the channel dimension, perform feature fusion, and construct a four-channel low-resolution input image. .

[0030] Step 2: Construct a back-end polarization image super-resolution network. The network module diagram is shown below. Figure 5 As shown, it includes: a shallow feature extraction module (SFE), a deep feature extraction module (DFE), a global feature fusion module (GFF), and a high-resolution image reconstruction module (HRR), and performs... The image was processed to obtain a super-resolution four-channel polarization image. Polarization image super-resolution network structure such as Figure 6 As shown.

[0031] Step 2.1, Shallow Feature Extraction (SFE) module... Perform a convolution operation to obtain the initial features. Then, on Perform a convolution operation to obtain shallow features. SFE is used to extract polarization information in low-resolution space; in this embodiment, the kernel size used in both convolution operations is 3×3.

[0032] Step 2.2: The Deep Feature Extraction (DFE) module consists of G residual dense blocks, where the g-th residual dense block is denoted as... Residual dense blocks Structure such as Figure 7 As shown, it includes: m convolutional units, a splicing layer, and a convolutional layer with a 1×1 kernel; where g∈[1,G], each convolutional unit consists of a convolutional layer with a 3×3 kernel and a LeakyReLU activation function layer; DFE performs dense connections and residual learning to enrich the hierarchical representation of low-resolution features; in this embodiment, the number of convolutional units m=8, and the number of residual dense blocks G=16.

[0033] When g=1, As the first Pre-sequence feature maps The g-th residual dense block is input and processed by m convolutional units to obtain m convolutional features. The m convolutional features are then concatenated along the channel dimension in a concatenation layer to obtain the g-th concatenated feature. This concatenated feature is then processed in a convolutional layer with a 1×1 kernel to obtain the g-th fused feature. Thus and Element-wise addition fusion is performed using residual joins to obtain the first... A dense feature map ; thus, the dense feature map output from the G residual dense blocks { After concatenating |g∈[1,G]} along the channel dimension, deep features are obtained. .

[0034] Step 2.3: The Global Feature Fusion (GFF) module consists of a concatenation layer, a 1×1 convolutional layer, and a 3×3 convolutional layer; [The following is a description of the process:] ...in sequence... After processing, the global residual features are obtained and compared with... After performing residual connections, global fusion features are obtained. GFF is used to globally fuse polarization features at different levels, making full use of low-resolution features at different levels.

[0035] Step 2.4: The High-Resolution Image Reconstruction (HRR) module consists of the Feature Refinement Module (FRB) and the Upsampling Module (UPS), and performs... After processing, a super-resolution four-channel polarization image is obtained. HRR is used to upsample the fused features and output a high-resolution polarization image.

[0036] Step 2.4.1 The input to the Feature Refinement (FRB) module first passes through a 5×5 convolutional layer and a Leaky ReLU activation function layer, then through a 3×3 convolutional layer and a Leaky ReLU activation function layer, to obtain the refined features. .

[0037] Step 2.4.2: Refine the features The image is input into the upsampling module UPS, and then processed sequentially through r subpixel convolutional units and a convolutional layer with a 3×3 kernel, outputting a super-resolution four-channel polarization image. Where r∈[1,R], each subpixel convolutional unit consists of a convolutional layer with a kernel of 3×3 and a subpixel convolutional layer; a subpixel convolutional unit can achieve 2x upsampling; in this embodiment, the number of subpixel convolutional units r=1.

[0038] Step 3, based on and Establish the total loss function of the back-end polarization image super-resolution reconstruction network. In this embodiment, the loss function consists of two parts: content loss and polarization-aware loss. Content loss focuses more on constraining the reconstructed high-resolution polarization image, ensuring that the reconstructed super-resolution polarization image is numerically consistent with the real high-resolution polarization image. Polarization-aware loss focuses more on reconstructing polarization information such as polarization degree image and polarization angle image, which can enhance the network's ability to learn polarization features, thereby achieving synchronous and accurate reconstruction of polarization information and overall image quality.

[0039] Step 3.1: Construct content loss using equation (9) : (9) In equation (9), represent Norm operations.

[0040] Step 3.2: Construct polarization sensing loss using equation (10) : (10) In equation (10), and These represent the extraction operators for the degree of polarization and the polarization angle, respectively. This represents the feature map extracted from the last convolutional layer before the first max pooling layer in the VGG-19 network.

[0041] Step 3.3: Construct the total loss function using equation (11) : (11) In equation (11), These are the trainable parameters for a polarization image super-resolution network. These are weighting coefficients; It is an empirical value whose function is to compensate for content loss. With polarization sensing loss The weights are assigned, and the values ​​of the two losses are constrained to be on the same order of magnitude. In this embodiment, It is 0.01.

[0042] Step 4: Train the polarization image super-resolution reconstruction network using the ADAM optimizer and calculate the total loss function. The network parameters are updated, and training stops when the set number of training iterations is reached, resulting in a trained polarization image super-resolution reconstruction model for super-resolution reconstruction of low-resolution polarization images. In this embodiment, the network is trained for 220 epochs during the training phase, with an initial learning rate set to... It decays by 0.5 every 20 epochs.

[0043] In this embodiment, an electronic device includes a memory and a processor. The memory stores a program that supports the processor in executing the above-described method, and the processor is configured to execute the program stored in the memory.

[0044] In this embodiment, a computer-readable storage medium stores a computer program, which is executed by a processor to perform the steps of the above method.

[0045] To evaluate the effectiveness of this method, different scenarios were selected in the dataset for validation. For objective evaluation, the standard image super-resolution metrics, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), were used to quantitatively evaluate the polarization image super-resolution algorithm.

[0046] Peak signal-to-noise ratio (PSNR) is used to evaluate the quality of the reconstructed super-resolution image by comparing it with the real high-resolution image. It is mainly used to quantify the pixel-level fidelity of the two. The higher the PSNR value, the higher the quality of the reconstructed super-resolution image. The PSNR is calculated using equation (12): (12) In equation (12), MAX represents the maximum possible value of an image pixel, and MSE represents the mean square error between the super-resolution image pixel and the real high-resolution image, as expressed in equation (13): (13) The structural similarity index measures the structural similarity between a super-resolution image and a real high-resolution image by considering brightness, contrast, and structure. It closely matches the intuitive perception of images by the human eye and is consistent with subjective visual quality assessment. Its value range is [0,1], and the closer the value is to 1, the higher the image similarity. The calculation formula is shown in equation (14): (14) In equation (14), , and These represent the similarity between the super-resolution image and the high-resolution image in terms of brightness, contrast, and structure, respectively. , and This represents the weight of different features when calculating the structural similarity index. When Then, the structural similarity index is obtained using equation (15). : (15) In equation (15), and These represent the mean and standard deviation of the image, respectively. and To maintain numerical stability, , , Indicates the dynamic range of image pixel values. and They are usually set to 0.01 and 0.03 respectively.

Claims

1. A method for super-resolution reconstruction of spatially modulated polarization images based on heterogeneous feature fusion, characterized in that, The procedure is as follows: Step 1: Obtain the interferometric intensity image with polarization information of the target scene, and construct a four-channel high-resolution ground truth image. and the corresponding four-channel low-resolution input image ; Step 2: Construct a back-end polarization image super-resolution network, including: a shallow feature extraction module (SFE), a deep feature extraction module (DFE), a global feature fusion module (GFF), and a high-resolution image reconstruction module (HRR), and then... The image was processed to obtain a super-resolution four-channel polarization image. ; Step 3, based on and Establish the total loss function of the back-end polarization image super-resolution reconstruction network. ; Step 4: Train the polarization image super-resolution reconstruction network using the ADAM optimizer and calculate the total loss function. The network parameters are updated, and training stops when the number of training iterations reaches a set number, thus obtaining a trained polarization image super-resolution reconstruction model, which is used for super-resolution reconstruction of low-resolution polarization images.

2. The spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion according to claim 1, characterized in that, Step 1 is performed as follows: Step 1.1: Demodulate the interference intensity image to obtain the Stokes parameter, which is used to calculate the polarization degree image and polarization angle image of the target scene; Step 1.2: Perform intensity mapping processing on the interference intensity image to obtain the original visual intensity image; Step 1.3: After normalizing the polarization degree image and polarization angle image respectively, we obtain the normalized polarization degree image and the normalized polarization angle image. Step 1.4: Perform feature fusion along the channel dimension on the four heterogeneous feature images: the original visual intensity image, the Stokes parameter total intensity image, the normalized polarization degree image, and the normalized polarization angle image, to construct a four-channel high-resolution ground truth image. ; Step 1.5: Downsample the original visual intensity image, the total intensity image of the Stokes parameter, the polarization degree image, and the polarization angle image respectively to obtain the low-resolution original visual intensity image R, the low-resolution total intensity image S0, the low-resolution polarization degree image, and the low-resolution polarization angle image. Step 1.6: After normalizing the low-resolution polarization degree image and the low-resolution polarization angle image respectively, we obtain the normalized low-resolution polarization degree image D and the normalized low-resolution polarization angle image A. Step 1.7: Perform feature fusion along the channel dimension for R, S0, D, and A to construct a four-channel low-resolution input image. .

3. The spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion according to claim 2, characterized in that, Step 2 is performed as follows: Step 2.1, Shallow Feature Extraction (SFE) module... Perform a convolution operation to obtain the initial features. Then, on Perform a convolution operation to obtain shallow features. ; Step 2.2: The deep feature extraction module (DFE) consists of G residual dense blocks, where the g-th residual dense block includes m convolutional units, a concatenation layer, and a convolutional layer with a 1×1 kernel; where g∈[1,G], each convolutional unit consists of a convolutional layer with a 3×3 kernel and a LeakyReLU activation function layer. When g=1, As the first Pre-sequence feature maps The input is placed into the g-th residual dense block and processed sequentially through m convolutional units to obtain m convolutional features; The m convolutional features are then concatenated along the channel dimension in a concatenation layer to obtain the g-th concatenated feature. This concatenated feature is then processed in a convolutional layer with a 1×1 kernel to obtain the g-th fused feature. Thus and Element-wise addition fusion is performed using residual joins to obtain the first... A dense feature map ; thus, the dense feature map output from the G residual dense blocks { After concatenating |g∈[1,G]} along the channel dimension, deep features are obtained. ; Step 2.3: The Global Feature Fusion (GFF) module consists of a concatenation layer, a 1×1 convolutional layer, and a 3×3 convolutional layer, which are then sequentially applied to... After processing, the obtained global residual features are then compared with... After performing residual connections, global fusion features are obtained. ; Step 2.4: The High-Resolution Image Reconstruction (HRR) module consists of the Feature Refinement Module (FRB) and the Upsampling Module (UPS), and performs... After processing, a super-resolution four-channel polarization image is obtained. .

4. The spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion according to claim 3, characterized in that, Step 2.4 is performed as follows: Step 2.4.1 The input to the Feature Refinement (FRB) module first passes through a 5×5 convolutional layer and a Leaky ReLU activation function layer, then through a 3×3 convolutional layer and a Leaky ReLU activation function layer, to obtain the refined features. ; Step 2.4.2: Refine the features The image is input into the upsampling module UPS, and then processed sequentially through r subpixel convolutional units and a convolutional layer with a 3×3 kernel, outputting a super-resolution four-channel polarization image. , where r∈[1,R], and each subpixel convolutional unit consists of a convolutional layer with a kernel of 3×3 and a subpixel convolutional layer.

5. The spatial modulation polarization image super-resolution reconstruction method based on heterogeneous feature fusion according to claim 4, characterized in that, Step 3 is performed as follows: Step 3.1: Construct content loss using equation (1) : (1) In equation (1), represent Norm operations; Step 3.2: Construct polarization sensing loss using equation (2) : (2) In equation (2), and These represent the extraction operators for the degree of polarization and the polarization angle, respectively. This represents the feature map extracted from the last convolutional layer before the first max pooling layer in the VGG-19 network. Step 3.3: Construct the total loss function using equation (3). : (3) In equation (3), The parameters to be trained for the polarization image super-resolution network are . These are the weighting coefficients.

6. An electronic device, comprising a memory and a processor, characterized in that, The memory is used to store a program that supports a processor in executing the method of any one of claims 1-5, the processor being configured to execute the program stored in the memory.

7. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the steps of the method according to any one of claims 1-5.

Images