Color polarization image demosaicing method

Abstract

The invention discloses a demosaicing method for a color polarization image. The demosaicing method comprises the following steps: acquiring original image data; preprocessing the acquired image data; constructing an initial color polarization image demosaicing network; training the initial color polarization image demosaicing network by using the loss function to obtain a color polarization image demosaicing network; and processing the to-be-processed image by using the color polarization image demosaicing network to obtain a demosaiced color polarization image and polarization information. According to the method, an initial color polarization image demosaicing network is constructed based on convolution sparse coding expansion, so that generalization and robustness of color polarization image demosaicing under a multi-noise condition are improved; meanwhile, a loss function with physical constraints is designed according to the polarization characteristic of the color polarization filtering array image, so that the physical consistency of the polarization degree and the polarization angle is kept, polarization artifacts are inhibited, and the polarization angle recovery precision is improved.

Description

Technical Field

[0001] This invention relates to the field of image reconstruction technology, and in particular to a method for demosaicing color polarized images. Background Technology

[0002] Polarization imaging technology can utilize the physical properties of the surface of the imaged object (such as geometry, surface roughness, and material composition) to modulate the polarization state of light, thereby achieving high-sensitivity perception of scene structure and material information. Therefore, it shows broad application prospects in fields such as target detection, image dehazing, and early diagnosis of medical lesions.

[0003] In recent years, the Color Polarization Filter Array (CPFA), an imaging method based on the Division of Focal Plane (DoFP) architecture, has achieved simultaneous acquisition of RGB three-channel information and multi-polarization direction information under a single exposure by integrating a miniature polarizer array with a color filter array (such as a Bayer array). This imaging method features high transmittance, high extinction ratio, and good real-time imaging capabilities, effectively meeting the polarization imaging requirements in dynamic scenes.

[0004] However, during the imaging process of a color polarization filter array, the originally high-dimensional color polarization information is compressed and mapped onto a two-dimensional mosaic structure. To obtain a color polarization image with full spatial resolution, polarization demosaic reconstruction must be performed on the original color polarization filter array data. For example... Figure 1 The diagram illustrates color polarization imaging based on a focal plane. Since the polarization demosaic process involves nonlinear calculations of polarization parameters, and the original observations do not meet benign conditions, this problem is essentially a nonlinear ill-conditioned problem. If the reconstruction strategy is poorly designed, it can easily introduce undesirable phenomena such as polarization artifacts, noise amplification, and physical inconsistencies into the reconstruction results.

[0005] Existing polarization demosaic methods based on interpolation or optimization can recover polarization information to some extent, but they generally suffer from limited reconstruction accuracy and high computational complexity, making it difficult to meet the real-time requirements of Division of Focal Plane (DoFP) cameras in practical applications. While end-to-end deep learning-based polarization demosaic methods proposed in recent years have achieved some improvement in polarization degree (DoP) reconstruction, the lack of physical constraints makes it difficult to effectively suppress polarization artifacts, leading to significant distortion of the polarization angle (AoP) structure information, thus affecting the reliability of subsequent downstream vision tasks. Summary of the Invention

[0006] The purpose of this invention is to provide a method for demosaicing of color polarized images, which has high reconstruction stability and can accurately recover the polarization information in the image.

[0007] The color polarization image de-mosaic method provided by this invention includes the following steps:

[0008] S1. Obtain the raw image data;

[0009] S2. Preprocess the acquired image data;

[0010] S3. Construct an initial color polarization image demosaic network, including: a front-end processing sub-network, a first Stokes calculation module, a first-stage processing sub-network, a second Stokes calculation module, and a second-stage processing sub-network;

[0011] The front-end processing sub-network is used to downsample the original image data, obtain its polarization information, and perform color interpolation on the downsampled result to obtain the input image of the first-stage processing sub-network.

[0012] The first Stokes calculation module is used to calculate the Stokes vector of the input image of the first-stage processing sub-network;

[0013] The first-stage processing sub-network is used to perform color image de-mosaic processing on the input image to generate an intermediate reconstructed image;

[0014] The second Stokes calculation module is used to calculate the Stokes vector of the intermediate reconstructed image;

[0015] The second-stage processing sub-network is used to perform polarization image de-mosaic processing on the intermediate reconstructed image to generate the final reconstructed image;

[0016] S4. Train the initial color polarization image demosaic network using the loss function to obtain the color polarization image demosaic network;

[0017] S5. Use a color polarization image desacrifice network to process the image to be processed, and obtain a desacrifice color polarization image.

[0018] The original image data is a complete color polarization image;

[0019] The preprocessing of the acquired image data refers to the degradation processing of the complete color polarized image to obtain a simulated original image.

[0020] The front-end processing sub-network described in step 3 includes a downsampling module, a color interpolation module, a gradient weight calculation module, and a noise generation module;

[0021] The output of the downsampling module is used as the input of the color interpolation module, and the output of the gradient weight calculation module is used as the input of the noise generation module.

[0022] The processing procedure of the front-end sub-network includes the following steps:

[0023] A1. Obtain the simulated original image and the complete color polarization image;

[0024] A2. The simulated original image is downsampled using a downsampling module to obtain Bayer images corresponding to each polarization angle;

[0025] A3. Perform color interpolation on the Bayer images at each polarization angle using the color interpolation module to obtain a simulated initial color polarization image;

[0026] A4. Through the gradient weight calculation module, calculate the gradient magnitude of the simulated original image, construct the corresponding edge intensity map, and further obtain the gradient weight map G;

[0027] A5. Using the noise generation module, the gradient weight map G is multiplied element-wise with the standard Gaussian noise map N to obtain a structured noise map N', where the standard Gaussian noise map N is the same size as the complete color polarization image; then the structured noise map N' is added to the simulated initial color polarization image to obtain the simulated noisy original image, which is used as the input image for the first-stage processing sub-network.

[0028] The first Stokes calculation module takes a simulated noisy original image as input and outputs the Stokes vector components of the simulated noisy original image. , ;

[0029] The second Stokes computation module takes the intermediate reconstructed image as input and outputs the Stokes vector components of the intermediate reconstructed image. , .

[0030] The first stage processing sub-network includes a first color convolution, a first color Stokes feature map module, a color convolution sparse coding unwinding module, a second color Stokes feature map module, a second color convolution, and a color convolution activation module;

[0031] The output of the first color convolution and the output of the color convolution activation module are used as the input of the first color Stokes feature mapping module. The output of the first color Stokes feature mapping module is used as the input of the color convolution sparse coding unrolling module. The output of the color convolution sparse coding unrolling module and the output of the color convolution activation module are used as the input of the second color Stokes feature mapping module. The output of the second color Stokes feature mapping module is used as the input of the second color convolution.

[0032] The processing procedure of the first-stage sub-network includes the following steps:

[0033] B1. Obtain the simulated noisy original image output by the front-end processing sub-network, and extract color features through the first color convolution;

[0034] B2. Obtain the Stokes vector components of the simulated noisy original image output by the first Stokes computation module. , The first Stokes feature is obtained by nonlinear mapping through a color convolutional activation module. The color convolutional activation module includes a concatenated convolutional layer and a SiLU activation function layer.

[0035] B3. The color features and the first Stokes features are fused through the first color Stokes feature mapping module to generate initial sparse features;

[0036] B4. The initial sparse features are processed layer by layer by the color convolutional sparse coding unwinding module to generate the corresponding first sparse features;

[0037] B5. The first sparse feature and the first Stokes feature are fused through the second color Stokes feature mapping module to generate the second sparse feature;

[0038] B6. The second sparse feature is processed by the second color convolution and then added to the simulated noisy original image to generate an intermediate reconstructed image.

[0039] The second-stage processing sub-network includes a polarization convolution sparse coding unrolling module, a polarization Stokes feature mapping module, a polarization convolution activation module, and a polarization convolution;

[0040] The input to the polarization convolution sparse coding unrolling module is the output of the second color Stokes feature mapping module. The output of the polarization convolution sparse coding unrolling module and the output of the second color Stokes feature mapping module are concatenated in the channel dimension, and together with the twice-channel feature output of the polarization convolution activation module, they are used as the input of the polarization Stokes feature mapping module. The output of the polarization Stokes feature mapping module is used as the input of the polarization convolution.

[0041] The second-stage processing of the sub-network includes the following steps:

[0042] C1. Obtain the second sparse feature output by the second color Stokes feature mapping module, and process the second sparse feature layer by layer through the polarization convolution sparse coding unpacking module to generate the corresponding third sparse feature.

[0043] C2. Concatenate the second and third sparse features to form a fourth sparse feature;

[0044] C3. Obtain the Stokes vector components of the intermediate reconstructed image output by the second Stokes computation module. , The corresponding second Stokes feature is generated through a polarization convolution activation module; the polarization convolution activation module is used to expand the number of channels to twice the original number and perform nonlinear mapping on the expanded features; the polarization convolution activation module includes a concatenated convolutional layer and a SiLU activation function layer.

[0045] C4. The fourth sparse feature and the second Stokes feature are fused through the polarization Stokes feature mapping module to generate the fifth sparse feature;

[0046] C5. The fifth sparse feature is integrated by polarization convolution to obtain the color polarization feature;

[0047] C6. Upsample the color polarization features to obtain the upsampled color polarization features;

[0048] C7. Upsample the simulated original image by a factor of 2 to obtain the upsampled simulated original image;

[0049] C8. Add the upsampled color polarization features to the upsampled simulated original image to obtain the final reconstructed image.

[0050] The first color Stokes feature mapping module, the second color Stokes feature mapping module, and the polarization Stokes feature mapping module all have the same structure.

[0051] The module's processing procedure includes the following steps:

[0052] D1. Obtain the input feature map and Stokes features;

[0053] D2. The input feature map is processed sequentially through a multi-depth convolutional head transpose attention module, a first normalization layer, a first mapping convolution, and a GeLU activation function layer to obtain the first intermediate mapping feature. The multi-depth convolutional head transpose attention module is the attention mechanism in Restormer.

[0054] D3. Stokes features are processed by the second normalization layer and the second mapping convolution to obtain the second intermediate mapping features;

[0055] D4. Multiply the first intermediate mapping feature by the second intermediate mapping feature;

[0056] D5. Add the multiplication result to the input feature map, process it through a gated feedforward network, and obtain the output of the Stokes feature map module. The gated feedforward network is the Restormer gated feedforward network.

[0057] In the first color Stokes feature mapping module, the input feature map is a color feature, and the Stokes feature is the first Stokes feature;

[0058] In the second color Stokes feature mapping module, the input feature map is the first sparse feature, and the Stokes feature is the first Stokes feature;

[0059] In the polarization Stokes feature mapping module, the input feature map is the fourth sparse feature, and the Stokes feature is the second Stokes feature.

[0060] Both the color convolutional sparse coding unwinding module and the polarization convolutional sparse coding unwinding module are composed of several convolutional dictionary units connected in series. The convolutional dictionary unit includes a first dictionary convolution, a second dictionary convolution, group normalization, and soft thresholding filtering.

[0061] The processing of the convolutional dictionary unit includes the following steps:

[0062] E1. Obtain the simulated noisy original image and the sparse features of the current cell input;

[0063] E2. Process the sparse features of the current unit input through first dictionary convolution;

[0064] E3. Subtract the features obtained by convolving the simulated noisy original image with the features obtained by the first dictionary;

[0065] E4. The subtraction result is processed by the second dictionary convolution, and the processed result is added to the sparse features input to the current unit;

[0066] E5. The summed features are then processed by group normalization and soft thresholding to obtain the sparse features of the current unit output.

[0067] In the first convolutional dictionary unit of the color convolutional sparse coding unwinding module, the current unit's input sparse feature is the initial sparse feature; in the first convolutional dictionary unit of the polarization convolutional sparse coding unwinding module, the current unit's input sparse feature is the second sparse feature; and in other convolutional dictionary units, the input sparse feature is the current unit's output sparse feature from the previous convolutional dictionary unit.

[0068] Step S4, which describes training the initial color polarization image demosaic network using a loss function, refers to training the first-stage processing subnetwork and the second-stage processing subnetwork in the initial color polarization image demosaic network.

[0069] The total loss function Calculate using the following formula:

[0070] In the formula, This is the total loss term for polarization physical fidelity. For the self-similar prior, the adaptive total variational loss term is used.

[0071] In the formula, This is the first polarization physical loss term, used to constrain the intermediate reconstructed image. This is the second polarization physical loss term, used to constrain the final reconstructed image;

[0072] In the formula, For adaptive total variation regularization term This is the self-similarity loss term;

[0073] The first polarization physical sub-loss term With the second polarization physics loss term All calculations are performed using the following formula:

[0074] In the formula, Choosing 1 or 2, 10 is used to balance the orders of magnitude. For pixel loss components, For gradient loss components, For structural similarity loss components;

[0075] In the formula, The value of the Stokes vector component S1 of the original image is used as the standard value for calculating the loss function. Choose 1 or 2. The value of the Stokes vector component S1 of the intermediate reconstructed image. The value of the Stokes vector component S2 of the original image is used as the standard value of S2 for calculating the loss function. The value of the Stokes vector component S2 of the intermediate reconstructed image. To simulate the polarization degree of the original image, the polarization degree is used as a standard value for calculating the loss function. The degree of polarization of the reconstructed image. To simulate the polarization angle of the original image, the standard value of the polarization angle is used to calculate the loss function. The polarization angle of the reconstructed image;

[0076] The value of the Stokes vector component S1 of the final reconstructed image. The value of the Stokes vector component S2 for the final reconstructed image. To ultimately reconstruct the polarization degree of the image, The polarization angle for the final reconstructed image;

[0077] For Sobel operators, To simulate the total light intensity of the original image, the total light intensity is used as the loss function. The standard value, Choose 1 or 2. The total light intensity of the reconstructed image is represented by the value of light. To simulate the Stokes vector of the original image, the standard value of the Stokes vector for calculating the loss function is used. The Stokes vector of the intermediate reconstructed image;

[0078] The total light intensity for the final reconstructed image, The Stokes vector for the final reconstructed image;

[0079] In the formula, It is a structural similarity index;

[0080] The adaptive total variation regularization term The following formula is used for calculation:

[0081] in,

[0082] In the formula, The total number of pixels in the image. The x-axis is... The vertical axis is , It is an exponent symbol.

[0083] The self-similarity loss term The following formula is used for calculation:

[0084] In the formula, Indicates the Kullback–Leibler divergence. To ultimately reconstruct the edge self-similarity weight map of the image, The edge self-similarity weight map of the complete color polarization image;

[0085] and All calculations are performed using the following formulas:

[0086] when superscript for Time represents the edge self-similarity weight of the final reconstructed image. , To search the region, traverse the center points of the image patches. It is the center of the search area. Let P be the vector representation of the center patch of the final reconstructed image, extracted with edge point P as the center. This is the vector representation of the matching block in the final reconstructed image that corresponds to the center block of the final reconstructed image. It is an exponential function. The scaling factor. For the number of channels, Image patch size,

[0087] when superscript for Time represents the edge self-similarity weight of the reference image. , This is the vector representation of the reference image center patch extracted with edge point P in the reference image as the center. A vector representation of the reference image matching block corresponding to the center block of the reference image;

[0088] The Obtained through the following steps:

[0089] F1. Based on the final reconstructed image, the gradient image of the final reconstructed image is calculated using the Laplace operator;

[0090] F2. Determine the gradient strength of each pixel in the gradient image of the final reconstructed image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 1; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the final reconstructed image. ;

[0091] F3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ;

[0092] F4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. ;

[0093] The Obtained through the following steps:

[0094] G1. Based on the complete color polarization image, calculate the gradient image of the reference image using the Laplace operator;

[0095] G2. Determine the gradient strength of each pixel in the gradient image of the reference image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 0; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the reference image. ;

[0096] G3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ;

[0097] G4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. .

[0098] Step S5 includes the following steps:

[0099] H1. Obtain the original image of the color polarization filter array to be processed;

[0100] H2. The front-end processing sub-network sequentially performs downsampling and color interpolation processing on the image to be processed to obtain an initial color polarization image;

[0101] H3. Calculate the Stokes vector of the initial color polarization image using the first Stokes calculation module;

[0102] H4. The initial color polarization image and the Stokes vector of the initial color polarization image are processed through the first-stage processing sub-network to obtain the first-stage sparse features and intermediate reconstructed image, and the first-stage sparse features and intermediate color polarization reconstructed image are output.

[0103] H5. Calculate the Stokes vector of the intermediate color polarization reconstructed image using the second Stokes calculation module;

[0104] H6. The Stokes vectors of the intermediate color polarization reconstructed image and the intermediate color polarization reconstructed image are processed by the second-stage processing sub-network to obtain the de-mosaic color polarization image.

[0105] The color polarization image demosaic method provided by this invention uses convolutional sparse coding (CSC) as the physical modeling basis and unfolds its iterative optimization process into a network structure of fixed depth, thereby improving the generalization ability and robustness of the color polarization image demosaic network under various noise conditions.

[0106] To address the polarization characteristics of Color Polarization Filter Array (CPFA) images, this invention designs a loss function with physical constraints to maintain the physical consistency between the degree of polarization (DoP) and the angle of polarization (AoP) and suppress polarization artifacts, thereby improving the AoP recovery accuracy.

[0107] Furthermore, during the training phase, structured noise modulated by gradient weights is superimposed on the input image, enabling the network to focus on learning the denoising and reconstruction capabilities of edge and structural regions. This enhances the model's adaptability to non-uniform noise and complex degradation without changing the network structure. Attached Figure Description

[0108] Figure 1 This is a schematic diagram of color polarization imaging based on the focal plane.

[0109] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0110] Figure 3 This is a schematic diagram of the structure of the color polarization image demosaic network of the method of the present invention.

[0111] Figure 4 This is a schematic diagram of the structure of the first-stage processing subnetwork of the method of the present invention.

[0112] Figure 5 This is a schematic diagram of the structure of the second-stage processing subnetwork of the method of the present invention.

[0113] Figure 6 This is a flowchart of the Stokes feature mapping module in the method of the present invention.

[0114] Figure 7 This is a flowchart illustrating the processing of the convolution dictionary unit in the method of the present invention.

[0115] Figure 8 This is a partial image of the OPID dataset in an embodiment of the method of the present invention.

[0116] Figure 9 This is the first visual comparison image of the method embodiment and the comparative example of the present invention on the comparison test set.

[0117] Figure 10 This is the second visual comparison image of the embodiment and the comparative example of the method of the present invention on the comparison test set. Detailed Implementation

[0118] like Figure 2 The diagram shown is a flowchart of the method of the present invention: The color polarization image demosaic method disclosed in this invention includes the following steps:

[0119] S1. Obtain the raw image data;

[0120] The original image data is a complete color polarization image;

[0121] S2. Preprocess the acquired image data;

[0122] The preprocessing of the acquired image data refers to the degradation processing of the complete color polarized image to obtain a simulated original image.

[0123] S3. Construct an initial color polarization image demosaic network, including: a front-end processing sub-network, a first Stokes calculation module, a first-stage processing sub-network, a second Stokes calculation module, and a second-stage processing sub-network;

[0124] A schematic diagram of the structure of a color polarization image demosaic network is shown below. Figure 3 As shown;

[0125] The front-end processing sub-network is used to downsample the original image data, obtain its polarization information, and perform color interpolation on the downsampled result to obtain the input image of the first-stage processing sub-network.

[0126] The first Stokes calculation module is used to calculate the Stokes vector of the input image of the first-stage processing sub-network;

[0127] The first-stage processing sub-network is used to perform color image de-mosaic processing on the input image to generate an intermediate reconstructed image;

[0128] The second Stokes calculation module is used to calculate the Stokes vector of the intermediate reconstructed image;

[0129] The second-stage processing sub-network is used to perform polarization image de-mosaic processing on the intermediate reconstructed image to generate the final reconstructed image;

[0130] The front-end processing sub-network includes a downsampling module, a color interpolation module, a gradient weight calculation module, and a noise generation module;

[0131] The output of the downsampling module is used as the input of the color interpolation module, and the output of the gradient weight calculation module is used as the input of the noise generation module.

[0132] The processing procedure of the front-end sub-network includes the following steps:

[0133] A1. Obtain the simulated original image and the complete color polarization image;

[0134] A2. The simulated original image is downsampled using a downsampling module to obtain Bayer images corresponding to each polarization angle;

[0135] A3. The Bayer images at each polarization angle are color interpolated using a color interpolation module to obtain a simulated initial color polarization image; preferably, the color interpolation is bilinear interpolation;

[0136] A4. The gradient magnitude of the simulated original image is calculated through the gradient weight calculation module, the corresponding edge intensity map is constructed, and the gradient weight map G is further obtained; preferably, the calculation of the gradient magnitude of the simulated original image refers to obtaining the edge intensity map of the simulated original image using the Sobel operator;

[0137] A5. Using the noise generation module, the gradient weight map G is multiplied element-wise with the standard Gaussian noise map N to obtain a structured noise map N', where the standard Gaussian noise map N is the same size as the complete color polarization image; then the structured noise map N' is obtained; the structured noise map N' is added to the simulated initial color polarization image to obtain the simulated noisy original image, which is used as the input image of the first-stage processing sub-network.

[0138] By calculating the gradient magnitude of the simulated original image to obtain a structured noise map N', a simulated noisy original image containing structured noise is constructed. Using this simulated noisy original image as training input, the network can focus on learning the denoising and reconstruction capabilities of edge and structure regions during training, thereby improving the model's adaptability to real-world complex noise scenes.

[0139] The first Stokes calculation module takes a simulated noisy original image as input and outputs the Stokes vector components of the simulated noisy original image. , ;

[0140] The second Stokes computation module takes the intermediate reconstructed image as input and outputs the Stokes vector components of the intermediate reconstructed image. , .

[0141] The first stage processing sub-network includes a first color convolution, a first color Stokes feature map module, a color convolution sparse coding unwinding module, a second color Stokes feature map module, a second color convolution, and a color convolution activation module;

[0142] like Figure 4 The diagram shown is a schematic of the first-stage processing subnetwork.

[0143] The output of the first color convolution and the output of the color convolution activation module are used as the input of the first color Stokes feature mapping module. The output of the first color Stokes feature mapping module is used as the input of the color convolution sparse coding unrolling module. The output of the color convolution sparse coding unrolling module and the output of the color convolution activation module are used as the input of the second color Stokes feature mapping module. The output of the second color Stokes feature mapping module is used as the input of the second color convolution.

[0144] The processing procedure of the first-stage sub-network includes the following steps:

[0145] B1. Obtain the simulated noisy original image output by the front-end processing sub-network, and extract color features through the first color convolution;

[0146] B2. Obtain the Stokes vector components of the simulated noisy original image output by the first Stokes computation module. , The first Stokes feature is obtained by nonlinear mapping through a color convolutional activation module. The color convolutional activation module includes a concatenated convolutional layer and a SiLU activation function layer.

[0147] B3. The color features and the first Stokes features are fused through the first color Stokes feature mapping module to generate initial sparse features;

[0148] B4. The initial sparse features are processed layer by layer by the color convolutional sparse coding unwinding module to generate the corresponding first sparse features;

[0149] B5. The first sparse feature and the first Stokes feature are fused through the second color Stokes feature mapping module to generate the second sparse feature;

[0150] B6. The second sparse feature is processed by the second color convolution and then added to the simulated noisy original image to generate an intermediate reconstructed image.

[0151] The second-stage processing sub-network includes a polarization convolution sparse coding unrolling module, a polarization Stokes feature mapping module, a polarization convolution activation module, and a polarization convolution;

[0152] like Figure 5 The diagram shows the structure of the second-stage processing subnetwork.

[0153] The input to the polarization convolution sparse coding unrolling module is the output of the second color Stokes feature mapping module. The output of the polarization convolution sparse coding unrolling module and the output of the second color Stokes feature mapping module are concatenated in the channel dimension, and together with the twice-channel feature output of the polarization convolution activation module, they are used as the input of the polarization Stokes feature mapping module. The output of the polarization Stokes feature mapping module is used as the input of the polarization convolution.

[0154] The second-stage processing of the sub-network includes the following steps:

[0155] C1. Obtain the second sparse feature output by the second color Stokes feature mapping module, and process the second sparse feature layer by layer through the polarization convolution sparse coding unpacking module to generate the corresponding third sparse feature.

[0156] C2. Concatenate the second and third sparse features to form a fourth sparse feature;

[0157] C3. Obtain the Stokes vector components of the intermediate reconstructed image output by the second Stokes computation module. , The corresponding second Stokes feature is generated through a polarization convolution activation module; the polarization convolution activation module is used to expand the number of channels to twice the original number and perform nonlinear mapping on the expanded features; the polarization convolution activation module includes a concatenated convolutional layer and a SiLU activation function layer.

[0158] C4. The fourth sparse feature and the second Stokes feature are fused through the polarization Stokes feature mapping module to generate the fifth sparse feature;

[0159] C5. The fifth sparse feature is integrated by polarization convolution to obtain the color polarization feature;

[0160] C6. Upsample the color polarization features to obtain the upsampled color polarization features;

[0161] C7. Upsample the simulated original image by a factor of 2 to obtain the upsampled simulated original image;

[0162] C8. Add the upsampled color polarization features to the upsampled simulated original image to obtain the final reconstructed image.

[0163] By inheriting and utilizing the sparse features of the color convolutional sparse coding unwinding module during the initialization and update phases, the polarization convolutional sparse coding unwinding module can fully leverage the contextual information of color features during the demosaicing process, thereby strengthening the correlation between color and polarization channels. By constructing a two-stage convolutional sparse coding structure prioritizing color and refining polarization, joint modeling of color and polarization information is achieved, thus reducing artifacts and cross-channel inconsistencies.

[0164] The first color Stokes feature mapping module, the second color Stokes feature mapping module, and the polarization Stokes feature mapping module all have the same structure.

[0165] like Figure 6 The diagram shows the processing flow of the Stokes feature mapping module.

[0166] The module's processing procedure includes the following steps:

[0167] D1. Obtain the input feature map and Stokes features;

[0168] D2. The input feature map is processed sequentially through a multi-depth convolutional head transpose attention module, a first normalization layer, a first mapping convolution, and a GeLU activation function layer to obtain the first intermediate mapping feature. The multi-depth convolutional head transpose attention module is the attention mechanism in Restormer.

[0169] D3. Stokes features are processed by the second normalization layer and the second mapping convolution to obtain the second intermediate mapping features;

[0170] D4. Multiply the first intermediate mapping feature by the second intermediate mapping feature;

[0171] D5. Add the multiplication result to the input feature map, and then process it through a gated feedforward network to obtain the output of the Stokes feature map module. The gated feedforward network is the gated feedforward network of Restormer.

[0172] In the first color Stokes feature mapping module, the input feature map is a color feature, and the Stokes feature is the first Stokes feature;

[0173] In the second color Stokes feature mapping module, the input feature map is the first sparse feature, and the Stokes feature is the first Stokes feature;

[0174] In the polarization Stokes feature mapping module, the input feature map is the fourth sparse feature, and the Stokes feature is the second Stokes feature.

[0175] Both the color convolutional sparse coding unwinding module and the polarization convolutional sparse coding unwinding module are composed of several convolutional dictionary units connected in series. The convolutional dictionary unit includes a first dictionary convolution, a second dictionary convolution, group normalization, and soft thresholding filtering.

[0176] like Figure 7 The diagram shows the processing flow of the convolution dictionary unit.

[0177] The processing of the convolutional dictionary unit includes the following steps:

[0178] E1. Obtain the simulated noisy original image and the sparse features of the current cell input;

[0179] E2. Process the sparse features of the current unit input through first dictionary convolution;

[0180] E3. Subtract the features obtained by convolving the simulated noisy original image with the features obtained by the first dictionary;

[0181] E4. The subtraction result is processed by the second dictionary convolution, and the processed result is added to the sparse features input to the current unit;

[0182] E5. The summed features are then processed by group normalization and soft thresholding to obtain the sparse features of the current unit output.

[0183] In the first convolutional dictionary unit of the color convolutional sparse coding unwinding module, the current unit's input sparse feature is the initial sparse feature; in the first convolutional dictionary unit of the polarization convolutional sparse coding unwinding module, the current unit's input sparse feature is the second sparse feature; and in other convolutional dictionary units, the input sparse feature is the current unit's output sparse feature from the previous convolutional dictionary unit.

[0184] The color convolutional sparse coding unrolling module and the polarization convolutional sparse coding unrolling module in this invention are built on the basis of convolutional sparse coding (CSC) for physical modeling. The derivation process is as follows:

[0185] The first step is to use convolutional sparse coding (CSC) to convert the image... Represented as a dictionary With sparsity coefficient The convolutional form is used to construct a model-driving term in the following form:

[0186] In the formula, D is a function representing the entire demosaic process. For weight parameters, The sparse regularization term is obtained through network learning rather than being manually set.

[0187] The second step, the iterative process, involves solving the CSC in the above equation using the Iterative Shrinkage-Thresholding Algorithm (ISA). The solution is then:

[0188] in,

[0189] In the formula, The sparse coefficients calculated for the current iteration unit. This indicates soft threshold filtering. The sparse coefficients are calculated from the previous iteration unit. Indicates the step size. It is a convolution operator used for mapping images to feature maps. It is a convolution operator used for feature mapping to image reconstruction.

[0190] The third step involves expanding the iterative optimization process of Convolutional Sparse Coding (CSC) into several layers of Convolutional Dictionary Units (CDUs), where each CDU corresponds to a specific iteration unit in the optimization process. Two convolutional layers are used to represent this. and Soft thresholding is equivalent to the ReLU activation function, therefore the convolutional dictionary unit is represented by the following formula:

[0191] In the formula, Output sparse features for the current cell. The activation function corresponds to the soft thresholding filter in the convolutional dictionary unit. This represents Group Normalization. The input sparse features are the current convolutional dictionary unit. For second dictionary convolution, Convolution of the first dictionary, To simulate the noisy original image,

[0192] Conv0 serves as both a dictionary and a decoder, while Conv1 transforms the original stride t into a learnable parameter, encoding the residual between the noisy original image and the decoder's output variables. This maps the image back to sparse features, which are then updated and subjected to nonlinear operations by a normalization layer and activation function. Each convolutional dictionary unit (CDU) completes one "gradient descent and sparse proximal" update, progressively optimizing the sparse features Z.

[0193] S4. Train the initial color polarization image demosaic network using the loss function to obtain the color polarization image demosaic network;

[0194] The training of the initial color polarization image demosaic network using the loss function refers to training the first-stage processing subnetwork and the second-stage processing subnetwork in the initial color polarization image demosaic network.

[0195] The total loss function Calculate using the following formula:

[0196] In the formula, This is the total loss term for polarization physical fidelity. For the self-similar prior, the adaptive total variational loss term is used.

[0197] In the formula, This is the first polarization physical loss term, used to constrain the intermediate reconstructed image. This is the second polarization physical loss term, used to constrain the final reconstructed image;

[0198] By applying this constraint to both the intermediate and final reconstructed images, the network can maintain a consistent reconstruction trend across all layers, thereby improving training stability.

[0199] In the formula, For adaptive total variation regularization term This is the self-similarity loss term;

[0200] The first polarization physical sub-loss term With the second polarization physics loss term All calculations are performed using the following formula:

[0201] In the formula, Choosing 1 or 2, 10 is used to balance the orders of magnitude. For pixel loss components, For gradient loss components, For structural similarity loss components;

[0202] In the formula, The value of the Stokes vector component S1 of the original image is used as the standard value for calculating the loss function. Choose 1 or 2. The value of the Stokes vector component S1 of the intermediate reconstructed image. The value of the Stokes vector component S2 of the original image is used as the standard value of S2 for calculating the loss function. The value of the Stokes vector component S2 of the intermediate reconstructed image. To simulate the polarization degree of the original image, the polarization degree is used as a standard value for calculating the loss function. The degree of polarization of the reconstructed image. To simulate the polarization angle of the original image, the standard value of the polarization angle is used to calculate the loss function. The polarization angle of the reconstructed image;

[0203] The value of the Stokes vector component S1 of the final reconstructed image. The value of the Stokes vector component S2 for the final reconstructed image. To ultimately reconstruct the polarization degree of the image, The polarization angle for the final reconstructed image;

[0204] By constraining polarization-related information such as S1, S2, DoP, and AoP at the pixel level, high-fidelity polarization information can be achieved.

[0205] For Sobel operators, To simulate the total light intensity of the original image, the total light intensity is used as the loss function. The standard value, Choose 1 or 2. The total light intensity of the reconstructed image is represented by the value of light. To simulate the Stokes vector of the original image, the standard value of the Stokes vector for calculating the loss function is used. The Stokes vector of the intermediate reconstructed image;

[0206] The total light intensity for the final reconstructed image, The Stokes vector for the final reconstructed image;

[0207] By constraining at the gradient level These variables, which do not involve nonlinear operations, can ensure basic edge information while avoiding the influence of noise.

[0208] In the formula, It is a structural similarity index;

[0209] The adaptive total variation regularization term The following formula is used for calculation: in,

[0210] In the formula, The total number of pixels in the image. The x-axis is... The vertical axis is , It is an exponent symbol.

[0211] While such global smoothing strategies can effectively suppress noise, they may also cause blurring of edge details under strong noise or low signal-to-noise ratio conditions. Therefore, in order to enhance the model's ability to perceive and reconstruct key polarization information, a self-similarity loss (SSL) is introduced to provide additional constraints in terms of local structure preservation.

[0212] The self-similarity loss term The following formula is used for calculation:

[0213] In the formula, Indicates the Kullback–Leibler divergence. To ultimately reconstruct the edge self-similarity weight map of the image, The edge self-similarity weight map of the complete color polarization image;

[0214] and All calculations are performed using the following formulas:

[0215] when superscript for Time represents the edge self-similarity weight of the final reconstructed image. , To search the region, traverse the center points of the image patches. It is the center of the search area. Let P be the vector representation of the center patch of the final reconstructed image, extracted with edge point P as the center. This is the vector representation of the matching block in the final reconstructed image that corresponds to the center block of the final reconstructed image. It is an exponential function. The scaling factor. For the number of channels, Image patch size,

[0216] when superscript for Time represents the edge self-similarity weight of the reference image. , This is the vector representation of the reference image center patch extracted with edge point P in the reference image as the center. A vector representation of the reference image matching block corresponding to the center block of the reference image;

[0217] The Obtained through the following steps:

[0218] F1. Based on the final reconstructed image, the gradient image of the final reconstructed image is calculated using the Laplace operator;

[0219] F2. Determine the gradient strength of each pixel in the gradient image of the final reconstructed image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 1; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the final reconstructed image. ;

[0220] F3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ;

[0221] F4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. ;

[0222] The Obtained through the following steps:

[0223] G1. Based on the complete color polarization image, calculate the gradient image of the reference image using the Laplace operator;

[0224] G2. Determine the gradient strength of each pixel in the gradient image of the reference image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 0; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the reference image. ;

[0225] G3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ;

[0226] G4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. .

[0227] S5. Process the image to be processed using a color polarization image desacrifice network to obtain a desacrifice color polarization image, including the following steps:

[0228] H1. Obtain the original image of the color polarization filter array to be processed;

[0229] H2. The front-end processing sub-network sequentially performs downsampling and color interpolation processing on the image to be processed to obtain an initial color polarization image;

[0230] H3. Calculate the Stokes vector of the initial color polarization image using the first Stokes calculation module;

[0231] H4. The initial color polarization image and the Stokes vector of the initial color polarization image are processed through the first-stage processing sub-network to obtain the first-stage sparse features and intermediate reconstructed image, and the first-stage sparse features and intermediate color polarization reconstructed image are output.

[0232] H5. Calculate the Stokes vector of the intermediate color polarization reconstructed image using the second Stokes calculation module;

[0233] H6. The Stokes vectors of the intermediate color polarization reconstructed image and the intermediate color polarization reconstructed image are processed by the second-stage processing sub-network to obtain the de-mosaic color polarization image.

[0234] The method of this invention constructs an initial color polarization image demosaic network based on convolutional sparse coding (CSC), which improves the generalization and robustness of color polarization image demosaic under multiple noise conditions. At the same time, a loss function with physical constraints is designed for the polarization characteristics of the color polarization filter array data to maintain the physical consistency of polarization degree (DoP) and polarization angle (AoP) and suppress polarization artifacts, thereby improving the accuracy of polarization angle (AoP) recovery.

[0235] The effects of the method of the present invention will be illustrated below with reference to an embodiment:

[0236] 1. Dataset preparation:

[0237] This embodiment supplements the public dataset by capturing several outdoor scenes, which are then named OPID. For example... Figure 8 The OPID dataset shown was constructed using a DoTP imaging system, capturing rich polarization variations at different times (dawn to dusk) and distances (near to far). Simultaneously, a DoFP camera was used to acquire real-world scene data, providing data under different indoor and outdoor lighting conditions for experimental verification.

[0238] This embodiment selects several publicly available datasets as the training and testing sets, namely the Monno+Qiu dataset and the PIDSR dataset, and augmented the datasets during both training and testing. For algorithm comparison, this embodiment selects five state-of-the-art color polarization image desamicing algorithms: IGRI2 based on interpolation strategies, NLCSR based on optimization models, TCPDNet, DCPM, and PIDSR based on deep learning. PIDSR is compared only for its desamicing results. To ensure fairness, all networks are retrained on the two aforementioned publicly available datasets.

[0239] 2. Comparative Experiment

[0240] For color polarized images at four angles, S0 and DoP, this embodiment uses PSNR and SSIM to evaluate image quality in comparative experiments. For AoP, because its dynamic range is [0, pi], this embodiment needs to use Mean Angular Error (MAE) to evaluate its accuracy. Furthermore, to evaluate the accuracy of the method on measured data, this embodiment also needs to introduce some no-reference metrics to evaluate the color polarized images; here, CLIPIQA is used to evaluate the overall quality of the S0 image. For example... Figure 9 and Figure 10 As shown, the visual effects of the proposed method (UP-Dem) compared with other existing methods on three DoTP simulation experimental images and one experimental image are illustrated, along with the S0, DoP, and AoP reconstructed by different methods. It is evident that the six algorithms do not show significant differences in S0; only in detail reproduction, IGRI2 and NLCSR exhibit some blurring. In DoP, the proposed method (UP-Dem) demonstrates a clear advantage in detail restoration and noise suppression, achieving the best preservation of weak texture edges. While NLCSR exhibits a relatively clean DoP, it cannot retain finer details, resulting in local oversmoothing. In AoP, the proposed method (UP-Dem) shows a more pronounced advantage in detail preservation. Compared to DoP, the calculation formula for AoP indicates that it is generally more difficult to restore to high accuracy. The fine textures of the two non-deep learning methods (AoP) have disappeared. The proposed method (UP-Dem) has the fewest artifacts and strong fidelity compared to the other three networks.

[0241] In the comparison of metrics, Table 1 shows that the method in this embodiment is optimal in almost all metrics of the three DoTP-based shooting datasets.

[0242] As can be seen from Table 2, the method in this embodiment has the best reference perception index in DoFP real-world shooting data, which quantitatively demonstrates the strong fidelity performance of the algorithm in this embodiment.

Claims

1. A method for demosaicing a color polarized image, comprising the following steps: S1. Obtain the raw image data; S2. Preprocess the acquired image data; S3. Construct an initial color polarization image demosaic network, including: a front-end processing sub-network, a first Stokes calculation module, a first-stage processing sub-network, a second Stokes calculation module, and a second-stage processing sub-network; The front-end processing sub-network is used to downsample the original image data, obtain its polarization information, and perform color interpolation on the downsampled result to obtain the input image of the first-stage processing sub-network. The first Stokes calculation module is used to calculate the Stokes vector of the input image of the first-stage processing sub-network; The first-stage processing sub-network is used to perform color image de-mosaic processing on the input image to generate an intermediate reconstructed image; The second Stokes calculation module is used to calculate the Stokes vector of the intermediate reconstructed image; The second-stage processing sub-network is used to perform polarization image de-mosaic processing on the intermediate reconstructed image to generate the final reconstructed image; S4. Train the initial color polarization image demosaic network using the loss function to obtain the color polarization image demosaic network; S5. Use a color polarization image desacrifice network to process the image to be processed, and obtain a desacrifice color polarization image.

2. The color polarization image demosaicking method of claim 1, wherein, The original image data is a complete color polarization image; The preprocessing of the acquired image data refers to the degradation processing of the complete color polarized image to obtain a simulated original image.

3. The method for de-mosaicing color polarized images according to claim 2, characterized in that, The front-end processing sub-network described in step 3 includes a downsampling module, a color interpolation module, a gradient weight calculation module, and a noise generation module; The output of the downsampling module is used as the input of the color interpolation module, and the output of the gradient weight calculation module is used as the input of the noise generation module. The processing procedure of the front-end sub-network includes the following steps: A1. Obtain the simulated original image and the complete color polarization image; A2. The simulated original image is downsampled using a downsampling module to obtain Bayer images corresponding to each polarization angle; A3. Perform color interpolation on the Bayer images at each polarization angle using the color interpolation module to obtain a simulated initial color polarization image; A4. Through the gradient weight calculation module, calculate the gradient magnitude of the simulated original image, construct the corresponding edge intensity map, and further obtain the gradient weight map G; A5. Using the noise generation module, the gradient weight map G is multiplied element-wise with the standard Gaussian noise map N to obtain a structured noise map N', where the standard Gaussian noise map N is the same size as the complete color polarization image; then the structured noise map N' is added to the simulated initial color polarization image to obtain the simulated noisy original image, which is used as the input image for the first-stage processing sub-network.

4. The method for de-mosaicing color polarized images according to claim 3, characterized in that, The first Stokes calculation module takes a simulated noisy original image as input and outputs the Stokes vector components of the simulated noisy original image. , ; The second Stokes computation module takes the intermediate reconstructed image as input and outputs the Stokes vector components of the intermediate reconstructed image. , .

5. The method for de-mosaicing color polarized images according to claim 4, characterized in that, The first stage processing sub-network includes a first color convolution, a first color Stokes feature map module, a color convolution sparse coding unwinding module, a second color Stokes feature map module, a second color convolution, and a color convolution activation module; The output of the first color convolution and the output of the color convolution activation module are used as the input of the first color Stokes feature mapping module. The output of the first color Stokes feature mapping module is used as the input of the color convolution sparse coding unrolling module. The output of the color convolution sparse coding unrolling module and the output of the color convolution activation module are used as the input of the second color Stokes feature mapping module. The output of the second color Stokes feature mapping module is used as the input of the second color convolution. The processing procedure of the first-stage sub-network includes the following steps: B1. Obtain the simulated noisy original image output by the front-end processing sub-network, and extract color features through the first color convolution; B2. Obtain the Stokes vector components of the simulated noisy original image output by the first Stokes computation module. , The first Stokes feature is obtained by nonlinear mapping through a color convolutional activation module. The color convolutional activation module includes a series of convolutional layers and a SiLU activation function layer. B3. The color features and the first Stokes features are fused through the first color Stokes feature mapping module to generate initial sparse features; B4. The initial sparse features are processed layer by layer by the color convolutional sparse coding unwinding module to generate the corresponding first sparse features; B5. The first sparse feature and the first Stokes feature are fused through the second color Stokes feature mapping module to generate the second sparse feature; B6. The second sparse feature is processed by the second color convolution and then added to the simulated noisy original image to generate an intermediate reconstructed image.

6. The method for demosaicing color polarized images according to claim 5, characterized in that, The second-stage processing sub-network includes a polarization convolution sparse coding unrolling module, a polarization Stokes feature mapping module, a polarization convolution activation module, and a polarization convolution; The input to the polarization convolution sparse coding unrolling module is the output of the second color Stokes feature mapping module. The output of the polarization convolution sparse coding unrolling module and the output of the second color Stokes feature mapping module are concatenated in the channel dimension, and together with the twice-channel feature output of the polarization convolution activation module, they are used as the input of the polarization Stokes feature mapping module. The output of the polarization Stokes feature mapping module is used as the input of the polarization convolution. The second-stage processing of the sub-network includes the following steps: C1. Obtain the second sparse feature output by the second color Stokes feature mapping module, and process the second sparse feature layer by layer through the polarization convolution sparse coding unpacking module to generate the corresponding third sparse feature. C2. Concatenate the second and third sparse features to form a fourth sparse feature; C3. Obtain the Stokes vector components of the intermediate reconstructed image output by the second Stokes computation module. , The corresponding second Stokes feature is generated through a polarization convolution activation module; the polarization convolution activation module is used to expand the number of channels to twice the original number and perform nonlinear mapping on the expanded features; the polarization convolution activation module includes a concatenated convolutional layer and a SiLU activation function layer. C4. The fourth sparse feature and the second Stokes feature are fused through the polarization Stokes feature mapping module to generate the fifth sparse feature; C5. The fifth sparse feature is integrated by polarization convolution to obtain the color polarization feature; C6. Upsample the color polarization features to obtain the upsampled color polarization features; C7. Upsample the simulated original image by a factor of 2 to obtain the upsampled simulated original image; C8. Add the upsampled color polarization features to the upsampled simulated original image to obtain the final reconstructed image.

7. The method for de-mosaicing color polarized images according to claim 6, characterized in that, The first color Stokes feature mapping module, the second color Stokes feature mapping module, and the polarization Stokes feature mapping module all have the same structure. The module's processing procedure includes the following steps: D1. Obtain the input feature map and Stokes features; D2. The input feature map is processed sequentially through a multi-depth convolutional head transpose attention module, a first normalization layer, a first mapping convolution, and a GeLU activation function layer to obtain the first intermediate mapping feature. The multi-depth convolutional head transpose attention module is the attention mechanism in Restormer. D3. Stokes features are processed by the second normalization layer and the second mapping convolution to obtain the second intermediate mapping features; D4. Multiply the first intermediate mapping feature by the second intermediate mapping feature; D5. Add the multiplication result to the input feature map, and then process it through a gated feedforward network to obtain the output of the Stokes feature map module. The gated feedforward network is the gated feedforward network of Restormer. In the first color Stokes feature mapping module, the input feature map is a color feature, and the Stokes feature is the first Stokes feature; In the second color Stokes feature mapping module, the input feature map is the first sparse feature, and the Stokes feature is the first Stokes feature; In the polarization Stokes feature mapping module, the input feature map is the fourth sparse feature, and the Stokes feature is the second Stokes feature.

8. The method for de-mosaicing color polarized images according to claim 7, characterized in that, Both the color convolutional sparse coding unwinding module and the polarization convolutional sparse coding unwinding module are composed of several convolutional dictionary units connected in series. The convolutional dictionary unit includes a first dictionary convolution, a second dictionary convolution, group normalization, and soft thresholding filtering. The processing of the convolutional dictionary unit includes the following steps: E1. Obtain the simulated noisy original image and the sparse features of the current cell input; E2. Process the sparse features of the current unit input through first dictionary convolution; E3. Subtract the features obtained by convolving the simulated noisy original image with the features obtained by the first dictionary; E4. The subtraction result is processed by the second dictionary convolution, and the processed result is added to the sparse features input to the current unit; E5. The summed features are then processed by group normalization and soft thresholding to obtain the sparse features of the current unit output. In the first convolutional dictionary unit of the color convolutional sparse coding unwinding module, the current unit's input sparse feature is the initial sparse feature; in the first convolutional dictionary unit of the polarization convolutional sparse coding unwinding module, the current unit's input sparse feature is the second sparse feature; and in other convolutional dictionary units, the input sparse feature is the current unit's output sparse feature from the previous convolutional dictionary unit.

9. The method for de-mosaicing color polarized images according to claim 8, characterized in that, Step S4, which describes training the initial color polarization image demosaic network using a loss function, refers to training the first-stage processing subnetwork and the second-stage processing subnetwork in the initial color polarization image demosaic network. The total loss function Calculate using the following formula: In the formula, This is the total loss term for polarization physical fidelity. For the self-similar prior, the adaptive total variational loss term is used. In the formula, This is the first polarization physical loss term, used to constrain the intermediate reconstructed image. This is the second polarization physical loss term, used to constrain the final reconstructed image; In the formula, For adaptive total variation regularization, This is the self-similarity loss term; The first polarization physical loss term With the second polarization physics loss term All calculations are performed using the following formula: In the formula, Choosing 1 or 2, 10 is used to balance the orders of magnitude. For pixel loss components, For gradient loss components, For structural similarity loss components; In the formula, The value of the Stokes vector component S1 of the original image is used as the standard value for calculating the loss function. Choose 1 or 2. The value of the Stokes vector component S1 of the intermediate reconstructed image. The value of the Stokes vector component S2 of the original image is used as the standard value of S2 for calculating the loss function. The value of the Stokes vector component S2 of the intermediate reconstructed image. To simulate the polarization degree of the original image, the polarization degree is used as a standard value for calculating the loss function. The degree of polarization of the reconstructed image. To simulate the polarization angle of the original image, the standard value of the polarization angle is used to calculate the loss function. The polarization angle of the reconstructed image; The value of the Stokes vector component S1 of the final reconstructed image. The value of the Stokes vector component S2 for the final reconstructed image. To ultimately reconstruct the polarization degree of the image, The polarization angle for the final reconstructed image; For Sobel operators, To simulate the total light intensity of the original image, the total light intensity is used as the loss function. The standard value, Choose 1 or 2. The total light intensity of the reconstructed image is represented by the value of light. To simulate the Stokes vector of the original image, the standard value of the Stokes vector for calculating the loss function is used. The Stokes vector of the intermediate reconstructed image; The total light intensity for the final reconstructed image, The Stokes vector for the final reconstructed image; In the formula, It is a structural similarity index; The adaptive total variation regularization term The following formula is used for calculation: in, In the formula, The total number of pixels in the image. The x-axis is... The vertical axis is , The symbol is for exponent. The self-similarity loss term The following formula is used for calculation: In the formula, Indicates the Kullback–Leibler divergence. To ultimately reconstruct the edge self-similarity weight map of the image, The edge self-similarity weight map of the complete color polarization image; and All calculations are performed using the following formulas: when superscript for Time represents the edge self-similarity weight of the final reconstructed image. , To search the region, traverse the center points of the image patches. It is the center of the search area. Let P be the vector representation of the center patch of the final reconstructed image, extracted with edge point P as the center. This is the vector representation of the matching block in the final reconstructed image that corresponds to the center block of the final reconstructed image. It is an exponential function. The scaling factor. For the number of channels, Image patch size, when superscript for Time represents the edge self-similarity weight of the reference image. , This is the vector representation of the reference image center patch extracted with edge point P in the reference image as the center. A vector representation of the reference image matching block corresponding to the center block of the reference image; The Obtained through the following steps: F1. Based on the final reconstructed image, the gradient image of the final reconstructed image is calculated using the Laplace operator; F2. Determine the gradient strength of each pixel in the gradient image of the final reconstructed image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 1; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the final reconstructed image. ; F3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ; F4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. ; The Obtained through the following steps: G1. Based on the complete color polarization image, calculate the gradient image of the reference image using the Laplace operator; G2. Determine the gradient strength of each pixel in the gradient image of the reference image. If the gradient strength of a pixel is greater than 0.1, then the pixel is set to 0; if the gradient strength of a pixel is less than or equal to 0.1, then the pixel is set to 0. This yields the binary edge mask corresponding to the reference image. ; G3. Extract binary edge mask The set is obtained by finding all the positions where the pixel value is 1. edge point ; G4. At the edge point The size of the building within the surrounding set range is The search area, within which the slider is sized Extract the matching block and its corresponding vector from the context window. .

10. The method for de-mosaicing a color polarized image according to claim 9, characterized in that, Step S5 includes the following steps: H1. Obtain the original image of the color polarization filter array to be processed; H2. The front-end processing sub-network sequentially performs downsampling and color interpolation processing on the image to be processed to obtain an initial color polarization image; H3. Calculate the Stokes vector of the initial color polarization image using the first Stokes calculation module; H4. The initial color polarization image and the Stokes vector of the initial color polarization image are processed through the first-stage processing sub-network to obtain the first-stage sparse features and intermediate reconstructed image, and the first-stage sparse features and intermediate color polarization reconstructed image are output. H5. Calculate the Stokes vector of the intermediate color polarization reconstructed image using the second Stokes calculation module; H6. The Stokes vectors of the intermediate color polarization reconstructed image and the intermediate color polarization reconstructed image are processed by the second-stage processing sub-network to obtain the de-mosaic color polarization image.

Images