Color polarization image demosaicing image fusion method based on deep learning

Abstract

The invention belongs to the technical field of polarization image processing, and particularly relates to a color polarization image demosaicing image fusion method based on deep learning, which comprises the following steps: S1, constructing a color polarization image demosaicing convolutional neural network model, and carrying out demosaicing processing on color polarization image data with missing pixel values to obtain a color polarization image demosaicing convolutional neural network model; and generating four panchromatic polarization images with different polarization angles, wherein the convolutional neural network model is composed of two sub-networks, namely a color demosaicing network and a polarization demosaicing network. The method can effectively solve the pixel interpolation problem of CPFA data of an existing color polarization image, improves the definition of the color polarization image, makes up for the technical defects of polarization imaging in a dynamic environment, provides the color polarization image based on the attention mechanism, analyzes the difference of imaging target information expressed by a color intensity image and a color polarization degree image, and improves the image quality. And an attention mechanism module is introduced in a targeted manner, so that the obtained fused image has better quality compared with a traditional fused image.

Description

Technical Field

[0001] The present invention relates to the technical field of polarization image processing, and in particular to a color polarization image demosaicing image fusion method based on deep learning. Background Art

[0002] Polarization is one of the important physical properties of light. When a target on the surface or in the atmosphere reflects, scatters, transmits, or radiates electromagnetic waves, it will generate specific polarization information determined by its own characteristics, and this polarization information can be used to analyze the shape, surface roughness, texture direction, and physical and chemical properties of the target. Polarization imaging technology can obtain multi-dimensional information such as the shape, material, and roughness of an object by analyzing the changes in polarization characteristics of light waves before and after they are reflected by an object, such as the degree of polarization and the polarization angle. In addition, in a weak light environment with scattering media, the ability to retain polarization information of reflected light is much greater than that of intensity information. Based on this, polarization imaging is widely used in target detection, flood disaster monitoring, communications, medical imaging, and other fields.

[0003] The polarization information of the target scene is mainly obtained by the polarization imaging system. There are two popular polarization imaging systems: time-sharing polarization imaging system and polarization imaging system. In the time-sharing polarization imaging system, the linear polarizer in front of the camera lens rotates in time sequence to obtain different polarization states of the pixel. However, the time-sharing polarimeter is only suitable for static scenes. In the split focal plane polarimeter, a micro-polarizer array (CPFA) is used to capture mosaic images of different polarization directions at one time, which can be used for dynamic scenes. However, each pixel of the captured mosaic image only contains information of one polarization direction, but demosaicing processing is required to generate images with different polarization angles.

[0004] According to the Stokes vector representation, the intensity image I and the degree of linear polarization (DoLP) image can be obtained from a series of images with different polarization angles. The intensity image with intensity information contains rich scene and texture information, but is easily affected by illumination. In contrast, the polarization image is not affected by the intensity of illumination. Even in poor lighting conditions or low contrast, it can record rich surface details of the object, such as object shape, surface roughness and curvature, which makes up for the defects of the intensity image. However, the polarization image does not contain scene information such as object color and scene illumination. Since the surface properties of the object do not change significantly, the surface texture will be lost in the polarization image. The complementarity of the two images prompts us to fuse them together. Therefore, there is a need in the prior art for a method that can combine color polarization image demosaicing and color polarization image fusion technology to improve the capture capability and image information amount of polarization images in dynamic scenes. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a color polarization image demosaicing image fusion method based on deep learning, which solves the problems raised in the above background technology.

[0007] (II) Technical solution

[0008] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0009] A color polarization image demosaicing image fusion method based on deep learning, comprising the following steps:

[0010] S1: Construct a color polarization image demosaicing convolutional neural network model, demosaic the color polarization image data with missing pixel values, and generate four full-color polarization images with different polarization angles. The convolutional neural network model consists of two sub-networks: a color demosaicing network and a polarization demosaicing network.

[0011] S2: Calculate the color intensity image I and the color degree of polarization image DoLP from the full-color polarization images at four different polarization angles;

[0012] S3: Construct a color polarization image fusion convolutional neural network model based on an attention mechanism, fuse the generated color I intensity image and the color polarization degree DoLP image, the convolutional neural network model includes an encoder module, a fusion module and a decoder module in sequence; the convolutional neural network model uses a loss function to evaluate the difference between the predicted value and the true value of the model, and the loss function is composed of a multi-scale weighted structural similarity loss and a perceptual loss weighted;

[0013] S4: training the convolutional neural network model described in S1 using a polarization image dataset collected by a focal plane color polarization imaging system;

[0014] S5: training the convolutional neural network model described in S3 using the demosaiced polarization image data processed by the color polarization image data demosaicing convolutional neural network described in S1;

[0015] S6: Input the test set image data into the trained convolutional neural network model and output the final fusion enhancement result.

[0016] Furthermore, in the color polarization image demosaicing convolutional neural network model in S1, the input CPFA original data is first arranged into four sub-sampled Bayer mosaic data according to four different polarization angles, and the height and width of the sub-sampled Bayer mosaic data are both half of the input original data, and then the four sub-sampled Bayer mosaic data are sent to the color demosaicing network;

[0017] Take the sub-sampling with a polarization degree of 90° as an example: Assume that the size of the input original data is H×W, and the size of the sub-sampled data is Specifically, assuming that the input CPFA data is I CPFA , four sub-sampled Bayer mosaic data are generated as follows:

[0018] Red channel: In the subsampled image, only the red channel pixels of the original data are retained;

[0019] R sub (i,j)=I CPFA (2i,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0020] Green1 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are even rows and odd columns;

[0021] G1 sub (i,j)=I CPFA (2i,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0022] Green2 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are odd rows and even columns;

[0023] G2 sub (i,j)=I CPFA (2i+1,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0024] Blue channel: In the subsampled image, only the blue channel pixels of the original data are retained;

[0025] B sub (i,j)=I CPFA (2i+1,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0026] In this way, the original CPFA data can be rearranged into four sub-sampled images with different polarization angles in the form of Bayer mosaic. Each sub-sampled image at a different polarization angle contains: sub (i,j), green channel 1 G1sub (i,j), green 2 channel G2 sub (i,j) and blue channel B sub (i,j);

[0027] In the color polarization image demosaicing convolutional neural network model, the color demosaicing network performs demosaicing on the four sub-sampled Bayer mosaic data to generate four sub-sampled RGB images, where the weights of the color demosaicing network are shared.

[0028] Furthermore, the subsampled RGB image generated by the color demosaicing network generates mosaic polarization data of the three RGB color channels using the same method as the pixel shuffling operation, which is then sent to the polarization demosaicing network. The polarization demosaicing network demosaicizes the three mosaic polarization data to generate a full RGB polarization image, forming an output 12-channel data, wherein the weights of the color demosaicing network are shared.

[0029] Furthermore, the color demosaicing network mainly consists of three parts: a channel attention mechanism module (SE Block), a bilinear interpolation operation and a refinement network; the polarization demosaicing network mainly consists of three parts: a spatial attention mechanism module (CBAM), a bilinear interpolation operation and a refinement network.

[0030] Furthermore, in the color polarization image demosaicing convolutional neural network model, the loss function is composed of the loss function L of the color demosaicing network C and the loss function L of the polarization demosaicing network CP Combination composition, the loss function L is calculated as follows

[0031]

[0032] L=L C +αL CP

[0033] Among them, for the color demosaicing network, the loss function L of the color demosaicing network is C , let U θ (θ∈Θ) is the process of subsampling the pixels with polarization direction θ of X to form single-channel subsampled Bayer CFA mosaic data. Let V θ It is the process of extracting a sub-sampled RGB image of polarization angle θ from the full color polarization image Z;

[0034] Among them, the loss function L for the polarization demosaicing network is CP , let Y(X;Φ)={g Φ (U θ (X))|θ∈Θ} is the output set of the color demosaicing network, let UC (c∈{R,G,B}) is the pixel shuffle operation of the Y(X;Φ) channel, let V C This is the process of extracting the c channel from the full-color polarization image Z; the loss weight parameter α=4.

[0035] Furthermore, the color intensity image and the color polarization image in S2 are solved by: decomposing the 12-channel full RGB polarization image according to the polarization angle to obtain four images with different polarization directions, which are 0°, 45°, 90° and 135° respectively, and then solving the I image and DoLP image according to the Stokes vector, and inputting them into the color polarization image fusion network. The specific formula is as follows:

[0036]

[0037] Where S 0 , I both represent the original light intensity information without biasing device, i.e., intensity image; S 1 , Q represents the difference between the horizontal polarization component and the vertical polarization component of the light wave; S 2 , U represents the difference between the polarization component of the light wave in the 45° direction and the polarization component in the 135° direction; S 3 , V represents the difference between the right-hand circular polarization component and the left-hand circular polarization component of the light wave; I R is the right-hand circular polarization component of the light wave; I L is the left-hand circularly polarized component of the light wave.

[0038] Furthermore, in the color polarization image fusion convolutional neural network model based on the attention mechanism in S3, the solved I image and DoLP image are first converted from RGB channels to YCrCb channels, and then the I image and DoLP image of the YCrCb channel are respectively input into the encoding module;

[0039] The encoder module is divided into two branches, which are used to input the intensity image and the polarization image respectively. Each branch includes the first convolution layer, the dense block module and the attention mechanism module in sequence. The data output of each layer is processed by the ReLU function. The first convolution layer consists of two 16-channel convolution layers. The intensity image and the polarization image are input into the two branches of the encoder network respectively, and the intensity feature map and the polarization feature map are output respectively.

[0040] The attention mechanism module in the intensity image encoder module branch is a spatial attention mechanism module. The structure of the spatial attention mechanism module is: input intensity feature map x i The feature descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. s (x) and g s(x), the two are concatenated and passed through the convolution layer to generate the spatial feature map α. Finally, the spatial feature map α is element-wise multiplied with the input feature map x to generate the output intensity feature map y i ;

[0041] The attention mechanism module in the polarization image encoder module branch is a channel attention mechanism module. The structure of the channel attention mechanism module is: input polarization feature map x dolp The spatial environment descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. c (x) and g c (x), the two are processed by the linear layer and the convolutional layer respectively, where the linear layer data output is processed by the ReLU function, and the processed data is element-wise added to generate the channel feature map β. Finally, the channel feature map β is element-wise multiplied with the input feature map x to generate the output polarization feature map y. dolp .

[0042] Further, the intensity feature map and the polarization degree feature map are input into a fusion module, and the intensity feature map and the polarization degree feature map are physically spliced ​​to obtain a fusion feature map. The fusion module is composed of four convolutional layers, and the data output by each convolutional layer is processed by a ReLU function;

[0043] The fused feature map is input into the decoder module, which includes four convolutional layers in sequence. The data output of the first three convolutional layers is processed by the ReLU function to output the YCrCb channel fused image.

[0044] The output image of the decoder module is physically spliced ​​with the intensity image, and then converted from YCrCb channels to RGB channels to generate the final fused image.

[0045] Furthermore, in the color polarization image fusion convolutional neural network model based on the attention mechanism, the loss function is calculated as follows:

[0046]

[0047] Loss=Loss MS-SSIM +λLoss perceptual

[0048] Among them, Loss MS-SSIM It is a loss function based on multi-scale weighted structural similarity. It is a loss function based on multi-scale weighted fusion quality index. w∈{3, 5, 7, 9, 11} represents different windows. SSIM(x, y; w) represents the structural similarity of two images under the w window. β w is the weight coefficient; Loss perceptual Feature-aware loss function based on pre-trained deep convolutional neural network, where I f is the input image, Itrue is the target image, Φ l (I f ) and Φ l (I true ) represent their feature representations at the lth layer in the pre-trained neural network, L is the number of feature layers, and λ is the balancing parameter.

[0049] Furthermore, the color polarization image dataset used for training the model in S4 is composed of two publicly available datasets: Tokyo Tech dataset and CPD Net dataset.

[0050] (III) Beneficial effects

[0051] Compared with the prior art, the present invention provides a color polarization image demosaicing image fusion method based on deep learning, which has the following beneficial effects:

[0052] The present invention can effectively solve the pixel interpolation (i.e., demosaicing) problem of existing color polarization image CPFA data, improve the clarity of color polarization images, and make up for the technical drawbacks of polarization imaging in dynamic environments. The proposed color polarization image based on attention mechanism analyzes the difference between color intensity image and color polarization image in expressing imaging target information, and introduces attention mechanism module in a targeted manner, so that the obtained fused image has better quality than the traditional fused image, laying a good foundation for better application in target detection, target tracking, etc.

[0053] By introducing the spatial attention mechanism module (CBAM) and channel attention mechanism module (SE Block) into the two sub-networks of color demosaicing and polarization demosaicing respectively according to different tasks, an attention learning process is formed, which improves the efficiency and quality of color polarization image demosaicing.

[0054] By analyzing the differences in the imaging target information expressed in the color intensity image and the color polarization image, a targeted attention mechanism module is designed and introduced to enhance the color and contrast information of objects in the intensity image, and enhance the detailed contour and surface texture information of objects in the polarization image, making the image details clearer than the original image, further improving the quality and information content of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a structural schematic diagram of the color polarization image demosaicing image fusion network model based on deep learning of the present invention;

[0056] Figure 2 It is a structural schematic diagram of the color polarization image demosaicing network model of the present invention;

[0057] Figure 3 It is a structural schematic diagram of the color polarization image fusion network model of the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Example

[0060] like Figure 1-3 As shown, a method for de-mosaicing and fusion of color polarization images based on deep learning is proposed in one embodiment of the present invention, comprising the following steps:

[0061] S1: Construct a color polarization image demosaicing convolutional neural network model to demosaic the color polarization image data with missing pixel values ​​and generate four full-color polarization images with different polarization angles. The convolutional neural network model consists of two sub-networks: a color demosaicing network and a polarization demosaicing network.

[0062] In the color polarization image demosaicing convolutional neural network model, the input CPFA raw data is first arranged into four sub-sampled Bayer mosaic data according to four different polarization angles. The height and width of the sub-sampled Bayer mosaic data are both half of the input raw data. Then, the four sub-sampled Bayer mosaic data are sent to the color demosaicing network.

[0063] Take the sub-sampling with a polarization degree of 90° as an example: Assume that the size of the input original data is H×W, and the size of the sub-sampled data is Specifically, assuming that the input CPFA data is I CPFA , four sub-sampled Bayer mosaic data are generated as follows:

[0064] Red channel: In the subsampled image, only the red channel pixels of the original data are retained;

[0065] R sub (i,j)=I CPFA (2i,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0066] Green1 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are even rows and odd columns;

[0067] G1 sub (i,j)=I CPFA (2i,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0068] Green2 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are odd rows and even columns;

[0069] G2 sub (i,j)=I CPFA (2i+1,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0070] Blue channel: In the subsampled image, only the blue channel pixels of the original data are retained;

[0071] B sub (i,j)=I CPFA (2i+1,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1]

[0072] In this way, the original CPFA data can be rearranged into four sub-sampled images with different polarization angles in the form of Bayer mosaic. Each sub-sampled image at a different polarization angle contains: sub (i,j), green channel 1 G1 sub (i,j), green 2 channel G2 sub (i,j) and blue channel B sub (i,j);

[0073] In the color polarization image demosaicing convolutional neural network model, the color demosaicing network performs demosaicing on the four sub-sampled Bayer mosaic data to generate four sub-sampled RGB images, where the weights of the color demosaicing network are shared;

[0074] The subsampled RGB image generated by the color demosaicing network is used to generate mosaic polarization data of the three RGB color channels in the same way as the pixel shuffle operation, which is then sent to the polarization demosaicing network. The polarization demosaicing network demosaicizes the three mosaic polarization data to generate a full RGB polarization image, forming an output 12-channel data, where the weights of the color demosaicing network are shared;

[0075] The color demosaicing network mainly consists of three parts: channel attention mechanism module (SE Block), bilinear interpolation operation and refinement network; the polarization demosaicing network mainly consists of three parts: spatial attention mechanism module (CBAM), bilinear interpolation operation and refinement network;

[0076] In the color polarization image demosaicing convolutional neural network model, the loss function is composed of the loss function L of the color demosaicing network C and the loss function L of the polarization demosaicing network CP Combination composition, the loss function L is calculated as follows

[0077]

[0078] L=L C +αL CP

[0079] Among them, for the color demosaicing network, the loss function L of the color demosaicing network is C , let U θ (θ∈Θ) is the process of subsampling the pixels with polarization direction θ of X to form single-channel subsampled Bayer CFA mosaic data. Let V θ It is the process of extracting a sub-sampled RGB image of polarization angle θ from the full color polarization image Z;

[0080] Among them, the loss function L for the polarization demosaicing network is CP , let Y(X;Φ)={g Φ (U θ (X))|θ∈Θ} is the output set of the color demosaicing network, let U C (c∈{R,G,B}) is the pixel shuffle operation of the Y(X;Φ) channel, let V C It is the process of extracting the c channel from the full color polarization image Z;

[0081] The loss weight parameter α=4.

[0082] S2: Calculate the color intensity image I and the color degree of polarization image DoLP from the full-color polarization images at four different polarization angles;

[0083] The color intensity image and color polarization image are calculated as follows: the 12-channel full RGB polarization image is decomposed according to the polarization angle to obtain four images with different polarization directions, which are 0°, 45°, 90° and 135° respectively. Then, the I image and DoLP image are calculated according to the Stokes vector and input into the color polarization image fusion network. The specific formula is as follows:

[0084]

[0085] Where S 0 , I both represent the original light intensity information without biasing device, i.e., intensity image; S 1 , Q represents the difference between the horizontal polarization component and the vertical polarization component of the light wave; S 2 , U represents the difference between the polarization component of the light wave in the 45° direction and the polarization component in the 135° direction; S 3 , V represents the difference between the right-hand circular polarization component and the left-hand circular polarization component of the light wave; I R is the right-hand circular polarization component of the light wave; I L is the left-hand circularly polarized component of the light wave.

[0086] S3: Construct a color polarization image fusion convolutional neural network model based on the attention mechanism, fuse the generated color I intensity image and color polarization degree DoLP image, the convolutional neural network model includes an encoder module, a fusion module and a decoder module in sequence; the convolutional neural network model uses a loss function to evaluate the difference between the predicted value and the true value of the model, and the loss function is composed of a multi-scale weighted structural similarity loss and a perceptual loss weighted;

[0087] In the color polarization image fusion convolutional neural network model based on the attention mechanism, the solved I image and DoLP image are first converted from RGB channels to YCrCb channels, and then the I image and DoLP image of the YCrCb channel are respectively input into the encoding module;

[0088] The encoder module is divided into two branches, which are used to input the intensity image and the polarization image respectively. Each branch includes the first convolution layer, the dense block module and the attention mechanism module in sequence. The data output of each layer is processed by the ReLU function. The first convolution layer consists of two 16-channel convolution layers. The intensity image and the polarization image are input into the two branches of the encoder network respectively, and the intensity feature map and the polarization feature map are output respectively.

[0089] The attention mechanism module in the intensity image encoder module branch is a spatial attention mechanism module. The structure of the spatial attention mechanism module is: input intensity feature map x i The feature descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. s (x) and g s (x), the two are concatenated and passed through the convolution layer to generate the spatial feature map α. Finally, the spatial feature map α is element-wise multiplied with the input feature map x to generate the output intensity feature map y i ;

[0090] The attention mechanism module in the polarization image encoder module branch is a channel attention mechanism module. The structure of the channel attention mechanism module is: input polarization feature map x dolpThe spatial environment descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. c (x) and g c (x), the two are processed by the linear layer and the convolutional layer respectively, where the linear layer data output is processed by the ReLU function, and the processed data is element-wise added to generate the channel feature map β. Finally, the channel feature map β is element-wise multiplied with the input feature map x to generate the output polarization feature map y. dolp ;

[0091] The intensity feature map and the polarization degree feature map are input into the fusion module, and the intensity feature map and the polarization degree feature map are physically spliced ​​to obtain the fusion feature map. The fusion module consists of four convolutional layers, and the data output by each convolutional layer is processed by the ReLU function;

[0092] The fused feature map is input into the decoder module, which includes four convolutional layers in sequence. The data output of the first three convolutional layers is processed by the ReLU function to output the YCrCb channel fused image.

[0093] Physically splice the output image of the decoder module with the intensity image, and then convert the YCrCb channel to the RGB channel to generate the final fused image;

[0094] In the color polarization image fusion convolutional neural network model based on the attention mechanism, the loss function is calculated as follows:

[0095]

[0096]

[0097] Loss=Loss MS-SSIM +λLoss perceptual

[0098] Among them, Loss MS-SSIM It is a loss function based on multi-scale weighted structural similarity. It is a loss function based on multi-scale weighted fusion quality index. w∈{3, 5, 7, 9, 11} represents different windows. SSIM(x, y; w) represents the structural similarity of two images under the w window. β w is the weight coefficient; Loss perceptual Feature-aware loss function based on pre-trained deep convolutional neural network, where I f is the input image, I true is the target image, Φ l (I f ) and Φ l (I true ) represent their feature representations at the lth layer in the pre-trained neural network, L is the number of feature layers, and λ is the balancing parameter.

[0099] S4: The convolutional neural network model of S1 is trained using a polarization image dataset collected by a focal plane color polarization imaging system. The color polarization image dataset used in the training model is composed of two publicly available datasets: Tokyo Tech dataset and CPD Net dataset.

[0100] S5: training the convolutional neural network model of S3 using the demosaiced polarization image data processed by the demosaiced convolutional neural network of the color polarization image data of S1;

[0101] S6: Input the test set image data into the trained convolutional neural network model and output the final fusion enhancement result.

[0102] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A color polarization image demosaicing image fusion method based on deep learning, characterized by: The following steps are involved: S1: Construct a color polarization image demosaicing convolutional neural network model, demosaic the color polarization image data with missing pixel values, and generate four full-color polarization images with different polarization angles. The convolutional neural network model consists of two sub-networks: a color demosaicing network and a polarization demosaicing network. S2: Calculate the color intensity image I and the color degree of polarization image DoLP from the full-color polarization images at four different polarization angles; S3: Construct a color polarization image fusion convolutional neural network model based on an attention mechanism, fuse the generated color I intensity image and the color polarization degree DoLP image, the convolutional neural network model includes an encoder module, a fusion module and a decoder module in sequence; the convolutional neural network model uses a loss function to evaluate the difference between the predicted value and the true value of the model, and the loss function is composed of a multi-scale weighted structural similarity loss and a perceptual loss weighted; S4: training the convolutional neural network model described in S1 using a polarization image dataset collected by a focal plane color polarization imaging system; S5: training the convolutional neural network model described in S3 using the demosaiced polarization image data processed by the color polarization image data demosaicing convolutional neural network described in S1; S6: Input the test set image data into the trained convolutional neural network model and output the final fusion enhancement result.

2. According to the deep learning-based color polarization image demosaicing image fusion method of claim 1, it is characterized by: In the color polarization image demosaicing convolutional neural network model in S1, the input CPFA original data is first arranged into four sub-sampled Bayer mosaic data according to four different polarization angles, and the height and width of the sub-sampled Bayer mosaic data are both half of the input original data, and then the four sub-sampled Bayer mosaic data are sent to the color demosaicing network; Take the sub-sampling with a polarization degree of 90° as an example: Assume that the size of the input original data is H×W, and the size of the sub-sampled data is Specifically, assuming that the input CPFA data is I CPFA , four sub-sampled Bayer mosaic data are generated as follows: Red channel: In the subsampled image, only the red channel pixels of the original data are retained; R sub (i,j)=I CPFA (2i,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1] Green1 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are even rows and odd columns; G1 sub (i,j)=I CPFA (2i,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1] Green2 channel: In the subsampled image, the green channel pixels of the original data are retained, and the positions are odd rows and even columns; G2 sub (i,j)=I CPFA (2i+1,2j), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1] Blue channel: In the subsampled image, only the blue channel pixels of the original data are retained; B sub (i,j)=I CPFA (2i+1,2j+1), where i∈[0,1,…,H sub -1],j∈[0,1,…,W sub -1] In this way, the original CPFA data can be rearranged into four sub-sampled images with different polarization angles in the form of Bayer mosaic. Each sub-sampled image at a different polarization angle contains: sub (i,j), green channel 1 G1 sub (i,j), green 2 channel G2 sub (i,j) and blue channel B sub (i,j); In the color polarization image demosaicing convolutional neural network model, the color demosaicing network performs demosaicing on the four sub-sampled Bayer mosaic data to generate four sub-sampled RGB images, where the weights of the color demosaicing network are shared.

3. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 2, characterized in that: The sub-sampled RGB image generated by the color demosaicing network generates mosaic polarization data of three RGB color channels using the same method as the pixel shuffling operation, which is then sent to the polarization demosaicing network. The polarization demosaicing network demosaicizes the three mosaic polarization data to generate a full RGB polarization image, forming an output 12-channel data, wherein the weights of the color demosaicing network are shared.

4. The method for de-mosaicing color polarization images according to claim 3, characterized in that: The color demosaicing network mainly consists of three parts: a channel attention mechanism module (SE Block), a bilinear interpolation operation and a refinement network; the polarization demosaicing network mainly consists of three parts: a spatial attention mechanism module (CBAM), a bilinear interpolation operation and a refinement network.

5. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 4, characterized in that: In the color polarization image demosaicing convolutional neural network model, the loss function is composed of the loss function L of the color demosaicing network C and the loss function L of the polarization demosaicing network CP Combination composition, the loss function L is calculated as follows L=L C +αL CP Among them, for the color demosaicing network, the loss function L of the color demosaicing network is C , let U θ (θ∈Θ) is the process of subsampling the pixels with polarization direction θ of X to form single-channel subsampled Bayer CFA mosaic data. Let V θ It is the process of extracting a sub-sampled RGB image of polarization angle θ from the full color polarization image Z; Among them, the loss function L for the polarization demosaicing network is CP , let Y(X;Φ)={g Φ (U θ (X))|θ∈Θ} is the output set of the color demosaicing network, let U C (c∈{R,G,B}) is the pixel shuffle operation of the Y(X;Φ) channel, let V C It is the process of extracting the c channel from the full color polarization image Z; The loss weight parameter α=4.

6. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 5, characterized in that: The color intensity image and color polarization degree image in S2 are solved by: decomposing the 12-channel full RGB polarization image according to the polarization angle to obtain four images with different polarization directions, which are 0°, 45°, 90° and 135° respectively, and then solving the I image and DoLP image according to the Stokes vector, and inputting them into the color polarization image fusion network. The specific formula is as follows: Where S0 and I represent the original light intensity information without a polarizing device, i.e., the intensity image; S1 and Q represent the difference between the horizontal polarization component and the vertical polarization component of the light wave; S2 and U represent the difference between the polarization component of the light wave at 45° and the polarization component at 135°; S3 and V represent the difference between the right-handed circular polarization component and the left-handed circular polarization component of the light wave; I R is the right-hand circular polarization component of the light wave; I L is the left-hand circularly polarized component of the light wave.

7. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 6, characterized in that: In the S3, in the color polarization image fusion convolutional neural network model based on the attention mechanism, the solved I image and DoLP image are first converted from the RGB channel to the YCrCb channel, and then the I image and DoLP image of the YCrCb channel are respectively input into the encoding module; The encoder module is divided into two branches, which are used to input the intensity image and the polarization image respectively. Each branch includes the first convolution layer, the dense block module and the attention mechanism module in sequence. The data output of each layer is processed by the ReLU function. The first convolution layer consists of two 16-channel convolution layers. The intensity image and the polarization image are input into the two branches of the encoder network respectively, and the intensity feature map and the polarization feature map are output respectively. The attention mechanism module in the intensity image encoder module branch is a spatial attention mechanism module. The structure of the spatial attention mechanism module is: input intensity feature map x i The feature descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. s (x) and g s (x), the two are concatenated and passed through the convolution layer to generate the spatial feature map α. Finally, the spatial feature map α is element-wise multiplied with the input feature map x to generate the output intensity feature map y i ; The attention mechanism module in the polarization image encoder module branch is a channel attention mechanism module. The structure of the channel attention mechanism module is: input polarization feature map x dolp The spatial environment descriptor f is generated by the average pooling layer and the maximum pooling layer respectively. c (x) and g c (x), the two are processed by the linear layer and the convolutional layer respectively, where the linear layer data output is processed by the ReLU function, and the processed data is element-wise added to generate the channel feature map β. Finally, the channel feature map β is element-wise multiplied with the input feature map x to generate the output polarization feature map y. dolp .

8. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 7, characterized in that: The intensity feature map and the polarization degree feature map are input into a fusion module, and the intensity feature map and the polarization degree feature map are physically spliced ​​to obtain a fusion feature map. The fusion module is composed of four convolutional layers, and the data output by each convolutional layer is processed by a ReLU function; The fused feature map is input into the decoder module, which includes four convolutional layers in sequence. The data output of the first three convolutional layers is processed by the ReLU function to output the YCrCb channel fused image. The output image of the decoder module is physically spliced ​​with the intensity image, and then converted from YCrCb channels to RGB channels to generate the final fused image.

9. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 8, characterized in that: In the color polarization image fusion convolutional neural network model based on the attention mechanism, the loss function is calculated as follows: Loss=Loss MS-SSIM +λLoss perceptual Among them, Loss MS-SSIM It is a loss function based on multi-scale weighted structural similarity. It is a loss function based on multi-scale weighted fusion quality index. w∈{3, 5, 7, 9, 11} represents different windows. SSIM(x, y; w) represents the structural similarity of two images under the w window. β w is the weight coefficient; Loss perceptual Feature-aware loss function based on pre-trained deep convolutional neural network, where I f is the input image, I true is the target image, Φ l (I f ) and Φ l (I true ) represent their feature representations at the lth layer in the pre-trained neural network, L is the number of feature layers, and λ is the balancing parameter.

10. The method for de-mosaicing and fusion of color polarization images based on deep learning according to claim 1, characterized in that: The color polarization image dataset used for training the model in S4 is composed of two publicly available datasets: Tokyo Tech dataset and CPD Net dataset.

Images