Improved color polarization image fusion method based on PFNET

Abstract

The invention discloses an improved color polarization image fusion method based on PFNET. Relates to the field of color polarization image fusion, in particular to the technical field of PFNET-based improved color polarization image fusion. According to the method, the integrity of feature expression is improved while color polarization image fusion is realized, and the method has high polarization resolution and rich textures. The method comprises the following steps: dividing a data set into a training set, a test set and a verification data set; improving a PFNET model: introducing an encoder module in a fusion image generator constructed by the shift window transformation feature extraction network into the PFNET model; a double-discriminator module constructed through a Gan adversarial network is introduced into the PFNET model; the total loss of the Gan adversarial network is introduced into the PFNET model; inputting the training set into the improved PFNET model to obtain a target model; inputting the test set into the target model to obtain an executable PFNET improved model; and inputting the verification data set into the executable PFNET improved model to obtain a fused image.

Description

Technical Field

[0001] The invention relates to the technical field of color polarization image fusion, and in particular to the technical field of color polarization image fusion based on PFNET improvement. Background Art

[0002] In today's technological development, color polarization image fusion technology has shown important application value in key areas such as remote sensing detection and target recognition. It provides strong technical support for many complex tasks and can help obtain more comprehensive and accurate image information.

[0003] However, most of the color polarization image fusion methods currently used are built on the traditional convolutional neural network (CNN) or basic generative adversarial network (GAN) architecture. Although these methods have achieved results to a certain extent, they still have some significant defects that need to be solved urgently: 1. Limited feature extraction capability: Traditional CNN mainly relies on local convolution operations, which makes it difficult to capture the global context information of the image. The lack of global information directly leads to the difficulty in integrating polarization characteristics (such as DoLP) and texture details in the fusion results, which seriously affects the quality of the fused image.

[0004] 2. Unbalanced information retention: Most existing GANs use a single discriminator design, which is prone to bias towards a certain type of source image during training, such as intensity images. Or linear polarization image. Once there is a deviation, it will cause the loss of complementary information, making it impossible for the fused image to retain the advantages of multi-source data at the same time, reducing the application value of the fused image.

[0005] 3. Limitations of loss functions: Traditional loss functions, represented by mean square error, focus too much on pixel-level alignment, but lack high-level semantic constraints such as structural similarity and visual fidelity. This causes the fused image to have problems such as low contrast and blurred edges, which greatly affects the visual effect and subsequent analysis of the image. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to propose a color polarization image fusion method based on PFNET improvement. By improving the PFNET model, the fusion of color polarization images is achieved while improving the integrity of feature expression, with both high polarization resolution and rich texture.

[0007] The method comprises the following steps: S1, obtaining a color polarization image dataset and preprocessing it, and dividing the preprocessed color polarization image dataset into a training set, a test set, and a validation set in a ratio of 6:2:2; S2. Improve the PFNET model, specifically: S21, replacing the original generator module with an improved generator module, wherein the improved generator module is obtained by introducing an encoder module in a fusion image generator constructed by a shift window transform feature extraction network into the original generator module of the PFNET model, and the improved generator module includes an upper encoder module and a lower encoder module; S22, introducing the dual discriminator module constructed by the Gan adversarial network into the PFNET model, the dual discriminator module includes discriminator 1 and discriminator 2; S23. The total loss of the Gan adversarial network is introduced into the loss function of the PFNET model to obtain an improved loss function. The improved loss function is specifically: ,in represents the total loss of the improved generator module, and denote the losses of discriminator 1 and discriminator 2 respectively; S3, input the training set into the improved PFNET model for iterative training. When the number of training times is greater than 100 and the total loss of the generator module and the loss of discriminator 1 and discriminator 2 no longer decrease significantly, the target model is obtained; S4, input the test set into the target model for testing, and obtain the performance test indicators: information entropy, standard deviation, visual information fidelity, structural similarity index and mutual information. When the performance test indicators reach a fixed threshold, an executable PFNET improved model is obtained; S5. Input the verification data set into the executable PFNET improved model to obtain the RGB fused image.

[0008] Furthermore, the preprocessing is specifically: using Stokes vector to describe the polarization characteristics of the color polarization image data set; the preprocessed color polarization image data set includes: intensity image and the degree of linear polarization image DOLP.

[0009] Furthermore, the uplink encoder module sequentially passes through the image segmentation module, the first stage layer module, the second stage layer module, the third stage layer module and the fourth stage layer module from input to output, and the structure of the downlink encoder module is consistent with that of the uplink encoder module; The first stage layer module includes: a linear embedding module and two shift window transformation modules; the second stage layer module and the fourth stage layer module both include: a block merging module and two shift window transformation modules; the third stage layer module includes: a block merging module and six shift window transformation modules.

[0010] Furthermore, the shift window transformation module passes through a multi-head self-attention module and a multi-layer perceptron module from input to output, and a normalization layer is set before each multi-head self-attention module and multi-layer perceptron module; each normalization layer is connected through a residual connection; The setting method of the multi-head self-attention module is: the multi-head self-attention module includes a window multi-head self-attention module and a shift window multi-head self-attention module; in several consecutive shift window transformation modules, from input to output, the multi-head self-attention module in the odd-numbered shift window transformation module is: the window multi-head self-attention module, and the multi-head self-attention module in the even-numbered shift window transformation module is: the shift window multi-head self-attention module.

[0011] Furthermore, the discriminator 1 passes through the first convolution module, the second convolution module, the third convolution module and the fully connected layer from input to output; the second convolution module and the third convolution module both use batch normalization layers; the first convolution module, the second convolution module and the third convolution module all use LeakyReLU activation functions; the fully connected layer uses tanh activation functions; the structure of discriminator 2 is consistent with that of discriminator 1.

[0012] Furthermore, the total loss of the improved generator module includes adversarial loss and content loss; the total loss calculation formula of the improved generator module is: ,in Represents resistance to loss, represents the training parameters, Indicates content loss.

[0013] Furthermore, during iterative training, the generator network learning rate of the improved PFNET model was set to 0.0001, the optimizer was AdamW, the dual discriminator network learning rate was 0.00001, the optimizer was Adam, the random rotation angle of the training set image was 5 degrees, the probability of random horizontal flipping was 5, and the probability of random vertical flipping was 5.

[0014] Furthermore, when the performance test index reaches a fixed threshold, specifically: Information entropy: 7.0, Standard Deviation: , Visual Information Fidelity: , Structural Similarity Index: And the mutual information: .

[0015] Furthermore, the implementation process of inputting the verification data set into the executable PFNET improved model to obtain the RGB fused image includes the following steps: S91, the linear polarization image DOLP and the intensity image in the training set Convert from RGB color space to YCrCb color space to obtain brightness information and Cr and Cb channel information; S92, the uplink encoder module and the downlink encoder module respectively extract features from the brightness information, and concatenate the feature information extracted by the uplink encoder module and the downlink encoder module to obtain brightness features; S93, the brightness feature is sequentially passed through a fusion module and a decoder module to obtain a reconstructed image; S94, inputting the reconstructed image and brightness information into discriminator 1 and discriminator 2 for adversarial training to obtain a feedback signal; S95 , splicing the reconstructed image with the Cr and Cb channel information, and converting the spliced ​​image into an RGB color space to obtain an RGB fused image.

[0016] Further, the feedback signal is used to optimize the improved generator module.

[0017] The beneficial effects of the method of the present invention are: 1. Realize deep fusion of global and local features: This invention introduces an encoder constructed by a shifted window transform feature extraction network (swintransformer), and uses the window multi-head self-attention (W-MSA) and shifted window multi-head self-attention (SW-MSA) mechanisms to achieve the coordinated operation of the two. This method can effectively extract the global polarization features and local texture details of the image, breaking the limitations of traditional methods in feature extraction, greatly improving the integrity of feature expression, providing more comprehensive and accurate feature information for subsequent image fusion, and significantly enhancing the ability of the fused image to describe complex scenes.

[0018] 2. Adversarial optimization based on dual discriminators: The present invention designs an independent dual discriminator module. During the training process, the two discriminators conduct adversarial training on the intensity image and the linear polarization image respectively. This unique design can effectively prompt the generator to evenly retain multi-source complementary information, effectively avoiding the problem of information bias. Through this method, it is ensured that the fused image not only has high polarization resolution and can accurately present information related to polarization characteristics, but also has rich texture, which greatly improves the quality and practicality of the fused image and meets the needs of multiple fields for high-quality fused images.

[0019] 3. Composite loss function: Improve the combined adversarial loss, structural similarity loss and mean square error loss, and comprehensively optimize the network from multiple dimensions such as pixel alignment, structural consistency and adversarial stability. Compared with traditional methods, the fused image trained based on this composite loss function performs better in key indicators such as information entropy (EN) and visual fidelity (VIF), and has significantly improved visual effects, image clarity, contrast and detail expression. At the information level, the image contains more information, which can provide a better data foundation for subsequent image analysis and applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flow chart of the method described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the PFNET improved model in an embodiment of the present invention; Figure 3 It is a schematic structural diagram of an uplink encoder module and a downlink encoder module in an improved generator module in an embodiment of the present invention; Figure 4 Schematic diagram of the structure of a continuous shift window transformation module in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] This embodiment provides a color polarization image fusion method based on PFNET improvement. The flowchart of the method is as follows: Figure 1 , the method comprises the following steps: S1. Obtain a color polarization image dataset and preprocess it. Divide the preprocessed color polarization image dataset into a training set, a test set, and a validation set in a ratio of 6:2:2.

[0023] The relevant operations of step S1 are introduced with a specific example: The color polarization image dataset includes color polarization images of different objects. In this embodiment, 40 color polarization images of the Tokyo Tech Color-Polarization Dataset are integrated into the training set to enhance the diversity of the training set data.

[0024] Natural light is non-polarized light, which becomes partially polarized light after being reflected by the surface of an artificial target. In this embodiment, Stokes vectors are used to describe the polarization characteristics of the color polarization image dataset. The preprocessed color polarization image data set includes: intensity image and the degree of linear polarization image DOLP.

[0025] The intensity image The calculation formula is: ,in represents the Stokes vector, represents the total light intensity, and Respectively represent the light intensity difference between the horizontal polarization component and the vertical polarization component, and Represents the intensity difference of 45° and 135° linear polarization components, and denote the intensity difference between right-handed and left-handed circularly polarized components, Respectively represent the light intensity values ​​of the polarization components at 0°, 45°, 90° and 135°, Represent the light intensity values ​​of right-handed and left-handed polarization components respectively; if , and If the value of any parameter is not equal to 0, it means that the incident light has polarization characteristics.

[0026] The calculation formula of the linear polarization degree image DOLP is: .

[0027] S2. Improve the PFNET model, specifically: S21, replacing the original generator module with an improved generator module, wherein the improved generator module is obtained by introducing an encoder module in a fusion image generator constructed by a shift window transform feature extraction network into the original generator module of the PFNET model, and the improved generator module includes an upper encoder module and a lower encoder module; S22, introducing the dual discriminator module constructed by the Gan adversarial network into the PFNET model, the dual discriminator module includes discriminator 1 and discriminator 2; S23. The total loss of the Gan adversarial network is introduced into the loss function of the PFNET model to obtain an improved loss function. The improved loss function is specifically: ,in represents the total loss of the improved generator module, and Denote the losses of Discriminator 1 and Discriminator 2, respectively.

[0028] The relevant operations of step S2 are introduced with a specific example: like Figure 3 As shown, the upper encoder module passes through the image block module (patchpartition), the first stage layer module, the second stage layer module, the third stage layer module and the fourth stage layer module from input to output; the structure of the lower encoder module is consistent with that of the upper encoder module.

[0029] The first-stage layer module includes: a linear embedding module (liner embedding) and two shift window transformation modules; the second-stage layer module and the fourth-stage layer module both include: a block merging module (patchmerging) and two shift window transformation modules; the third-stage layer module includes: a block merging module and six shift window transformation modules.

[0030] The role of the encoder in the improved generator module is to extract features from the input image and capture more advanced features by increasing the number of channels (depth). Therefore, the quality of feature extraction affects the quality of subsequent fused image generation. Introducing the shifted window transform feature extraction network to construct the encoder module of the fused image generator can enhance the ability of the fused image generator to extract local and global information from the image, thereby improving the quality of the fused image generated by the generator.

[0031] like Figure 3 and 4 As shown in the figure, the shift window transformation module passes through the multi-head self-attention module (MSA) and the multi-layer perceptron module (MLP) from input to output, and a normalization layer (LN) is set before each multi-head self-attention module and multi-layer perceptron module; each normalization layer is connected through a residual connection.

[0032] The setting method of the multi-head self-attention module is as follows: the multi-head self-attention module includes a window multi-head self-attention module and a shifted window multi-head self-attention module; in several consecutive shifted window transformation modules, from input to output, the multi-head self-attention module in the odd-numbered shifted window transformation module is: a window multi-head self-attention module (Windows Multi-Head Self-Attention, WMSA), and the multi-head self-attention module in the even-numbered shifted window transformation module is: a shifted window multi-head self-attention module (Shifted Windows Multi-Head SelfAttention, SW-MSA).

[0033] The image input to the shift window transform module must first be layer normalized and then input into the multi-head self-attention module. During this process, the attention weights between different positions will be adjusted according to the pre-calculated attention mask to ensure that only elements in the same window can pay attention to each other. The new features obtained above are added to the initial input features to form a residual connection, which is then passed through a normalization layer and then sent to the multi-layer perception module (MLP) for further transformation; the multi-layer perception module usually contains two linear layers and a GELU activation function to enhance nonlinear expression capabilities. The output of the multi-layer perception module is also added with another residual connection, that is, it is added to the normalized input features, and finally input into the next module as the output signal of the module.

[0034] When the multi-head self-attention module calculates self-attention, the input feature sequence is mapped into three different feature representations, namely Q, K and V, where Q, K and V represent the query matrix, key matrix and value matrix respectively, specifically: , , ,in represents the input sequence, Represents a learnable parameter; the calculation formula for self-attention is as follows: , where Z represents the learnable relative position encoding, represents the dimension of K, represents the transpose of K.

[0035] The role of the discriminator in the image fusion network is to evaluate the image fusion quality and provide feedback signals to help the generator improve its output to produce a fused image that fully integrates the information of the two source images. The Gan adversarial network is introduced to construct a dual discriminator module. Through adversarial training, the fused image generated by the improved generator module is forced to retain more source image information, so that the generated fused image has the complementary information of the two source images at the same time, realizing the fusion of complementary information.

[0036] like Figure 2 As shown, the discriminator 1 passes through the first convolution module, the second convolution module, the third convolution module and the fully connected layer from input to output; the second convolution module and the third convolution module both use the batch normalization layer; the first convolution module, the second convolution module and the third convolution module all use LeakyReLU Activation function; the fully connected layer uses tanh Activation function; The structure of Discriminator 2 is consistent with Discriminator 1.

[0037] LeakyReLU The calculation formula of the activation function is: ,in, Indicates that the input can be any real number. Represents the leakage coefficient, which has a value between 0 and 1. The value is 0.01.

[0038] Discriminator 1 is used to distinguish the fused image from the first source image, and discriminator 2 is used to distinguish the fused image from the second source image. Taking discriminator 1 as an example: the input of discriminator 1 passes through the first convolution module and outputs an output tensor with 16 channels. The second convolution module receives the output of the first convolution module and outputs an output tensor with 32 channels. The third convolution module receives the output of the second convolution module and outputs an output tensor with 64 channels. The fully connected layer uses tanh The activation function generates a scalar to estimate the probability that the input image comes from the first class of images rather than the fused image, specifically: , where t means the input can be any real number, and e means a natural number; The role of the image fusion network loss function is to define the difference between the model's predicted output and the expected output, and to guide the model on how to adjust its parameters to minimize this difference. Here, the improvement of the PFNET image fusion network loss function by combining the improved total loss of the generator module with the dual discriminator loss helps to improve the quality of the fused image and stabilize the training process to achieve the best fusion effect.

[0039] The total loss of the improved generator module includes adversarial loss and content loss; the calculation formula of the total loss of the improved generator module is: ,in, Represents resistance to loss, Indicates content loss, Represents training parameters. In this embodiment Set to 2.

[0040] Said The calculation formula is: ,in, represents the mathematical expectation, represents the fused image containing only brightness information generated by the improved generator module, represents discriminator 1, Denotes discriminator 2.

[0041] Said The calculation formula is: ,in represents the structural similarity loss, , represents the harmonic coefficient. In this embodiment Set to 0.01.

[0042] Structural similarity It is part of the most widely used content loss function in the field of image fusion, which reflects the similarity between images in terms of light intensity, contrast and structure. The calculation formula is: ,in, It means calculating the similarity between two images. and Respectively represent the brightness information images of source image 1 and source image 2, and Represents a weight parameter, which is used to balance the contribution of the two structural similarity indexes (SSIM) in the total loss function Lssim. and All are set to 0.5; Represents the mean absolute error loss. It can directly reflect the average difference between the predicted value and the true value. The calculation formula is: , where M and N represent the height and width of the image respectively, Represents intensity image and the average value of the linear polarization image DOLP at the pixel level. The intensity image and the degree of linear polarization image DOLP are complementary information of the same scene, so the intensity image And the linear polarization image DOLP Taking the average value will help to fuse the results.

[0043] The fused image generated by the improved generator module includes information from two different source images. Therefore, using only one discriminator may be biased towards images from a certain source, resulting in the loss of some information. Therefore, we constructed a dual discriminator module through the Gan adversarial network. Discriminator 1 and Discriminator 2 calculate the discriminator loss separately to achieve better preservation of source image information. Discriminator 1 and Discriminator 2 are independent of each other but use the same loss function. The loss function calculation formula of a single discriminator is: .

[0044] in, Representation Discriminator , Represents the source image Brightness information, , and Represents the source image and the source image .

[0045] S3. Input the training set into the improved PFNET model for iterative training. When the number of training times is greater than 100 and the total loss of the generator module and the loss of discriminator 1 and discriminator 2 no longer decrease significantly, the target model is obtained.

[0046] The relevant operations of step S3 are introduced with a specific example: During iterative training, the generator network learning rate of the improved PFNET model was set to 0.0001, the optimizer was AdamW, the dual discriminator network learning rate was 0.00001, the optimizer was Adam, the random rotation angle of the training set image was 5 degrees, the probability of random horizontal flipping was 5, and the probability of random vertical flipping was 5.

[0047] S4. Input the test set into the target model for testing and obtain performance test indicators: information entropy, standard deviation, visual information fidelity, structural similarity index and mutual information. When the performance test indicators reach a fixed threshold, an executable PFNET improved model is obtained.

[0048] The relevant operations of step S4 are introduced with a specific example: When the performance test index reaches a fixed threshold, it is specifically: Information entropy: 7.0, Standard Deviation: , Visual Information Fidelity: , Structural Similarity Index: And the mutual information: .

[0049] Information Entropy It is a non-reference evaluation index based on information theory, which is used to measure the amount of information contained in the fused image. The calculation formula of EN is as follows: , where L represents the grayscale level of the image, Indicates Normalized histogram of gray levels in the fused image.

[0050] Standard Deviation represents the distribution and contrast of pixel values ​​in an image, The larger the value, the higher the contrast. Standard deviation The calculation formula is as follows: , where M and N represent the height and width of the image respectively, represents the mean value of the image pixel values, Indicates that the image is at position Pixel value Visual information fidelity Consistent with the human visual system, the information fidelity can be measured and calculated through four steps: (1) filter and separate the source image and the fused image into different blocks; (2) evaluate the visual information of each block; (3) calculate the VIF of each sub-band; (4) calculate the overall metric. The visual information fidelity is proportional to the performance of the fusion method.

[0051] Structural Similarity Index It is used to measure the similarity between two images in terms of correlation, brightness, and contrast distortion. Specifically: ,in, Display Window Size The local similarity between images x and y, Represents image x in window Part of the image inside, and Respectively The mean and variance of express and The covariance of Indicates that image y is in the window Part of the image inside, and Respectively The mean and variance of and To maintain stability, in this embodiment, and Set to Hewei .

[0052] In polarization image fusion, SSIM consists of the structural similarity between the two source images and the fused image. Structural Similarity Index The calculation formula is as follows: .

[0053] in, and Represents the weight coefficient, used for balancing Images and The contribution of the image to the overall structural similarity index, in this embodiment, and Both are set to 0.5.

[0054] Mutual Information A method used to measure the similarity between the fused image and the source image information. The larger the mutual information value, the more source image information is contained in the fused image, and the better the fusion effect is. The calculation formula is as follows: ,in, Represents the mutual information function formula, W includes source image A and source image B, represents the fused image, Represents the source image and fused image The joint histogram of and For source image and fused image The marginal histogram statistical probability, represents the pixel intensity value (or grayscale value) of the source image W, Represents the pixel intensity value (or grayscale value) of the fused image F.

[0055] S5. Input the verification data set into the executable PFNET improved model to obtain the RGB fused image.

[0056] The relevant operations of step S5 are introduced with a specific example: The implementation process of inputting the verification data set into the executable PFNET improved model to obtain the RGB fused image includes the following steps: S91, the linear polarization image DOLP and the intensity image in the training set Convert from RGB color space to YCrCb color space to obtain brightness information and Cr and Cb channel information; S92, the upper encoder module and the lower encoder module respectively extract features from the brightness information. In the upper encoder module and the lower encoder module, the image block module divides the input image into image blocks of 4×4 size, and converts the minimum operation unit of the image from pixels to image blocks. The image processed by the image block module passes through the first stage layer, and the image blocks of 4×4 size are embedded into the vector. Subsequently, the image processed by the first stage layer passes through the second, third and fourth stage layers in sequence. In the second, third and fourth stage layers, each time a stage layer is passed, the size of the input image becomes half of the original size, and the feature dimension becomes twice of the original size. The feature information extracted by the upper encoder module and the lower encoder module is spliced ​​to obtain brightness features. The image block module is used to reduce the size of the input image while realizing the cascading of the image blocks in the feature dimension. S93, the brightness feature is sequentially passed through a fusion module and a decoder module to obtain a reconstructed image of a fused image containing only brightness information; S94, inputting the reconstructed image and brightness information into discriminator 1 and discriminator 2 for adversarial training to obtain a feedback signal; S95 , splicing the reconstructed image with the Cr and Cb channel information, and converting the spliced ​​image into an RGB color space to obtain an RGB fused image.

[0057] like Figure 2 As shown, the fusion module passes through two convolution and ReLU activation function layers from input to output, and each convolution layer uses the ReLU activation function; the decoder module passes through five convolution layers from input to output, and the first four convolution layers use the ReLU activation function.

[0058] The feedback signal is used to optimize the improved generator module: by adjusting the parameters (adversarial loss) to guide the generator to improve the generator generated images.

Claims

1. A color polarization image fusion method based on PFNET improvement, characterized in that: The method comprises the following steps: S1, obtaining a color polarization image dataset and preprocessing it, and dividing the preprocessed color polarization image dataset into a training set, a test set, and a validation set in a ratio of 6:2:2; S2. Improve the PFNET model, specifically: S21, replacing the original generator module with an improved generator module, wherein the improved generator module is obtained by introducing an encoder module in a fusion image generator constructed by a shift window transform feature extraction network into the original generator module of the PFNET model, and the improved generator module includes an upper encoder module and a lower encoder module; S22, introducing the dual discriminator module constructed by the Gan adversarial network into the PFNET model, the dual discriminator module includes discriminator 1 and discriminator 2; S23. The total loss of the Gan adversarial network is introduced into the loss function of the PFNET model to obtain an improved loss function. The improved loss function is specifically: ,in represents the total loss of the improved generator module, and denote the losses of discriminator 1 and discriminator 2 respectively; S3, input the training set into the improved PFNET model for iterative training. When the number of training times is greater than 100 and the total loss of the generator module and the loss of discriminator 1 and discriminator 2 no longer decrease significantly, the target model is obtained; S4, input the test set into the target model for testing, and obtain the performance test indicators: information entropy, standard deviation, visual information fidelity, structural similarity index and mutual information. When the performance test indicators reach a fixed threshold, an executable PFNET improved model is obtained; S5. Input the verification data set into the executable PFNET improved model to obtain the RGB fused image.

2. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: The preprocessing is specifically: using Stokes vector to describe the polarization characteristics of the color polarization image data set; the preprocessed color polarization image data set includes: intensity image and the degree of linear polarization image DOLP.

3. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: The on-line encoder module passes through the image segmentation module, the first stage layer module, the second stage layer module, the third stage layer module and the fourth stage layer module in sequence from input to output; The structure of the downlink encoder module is consistent with that of the uplink encoder module; The first stage layer module includes: a linear embedding module and two shift window transformation modules; the second stage layer module and the fourth stage layer module both include: a block merging module and two shift window transformation modules; the third stage layer module includes: a block merging module and six shift window transformation modules.

4. The color polarization image fusion method based on PFNET improvement according to claim 3 is characterized in that: The shift window transformation module passes through a multi-head self-attention module and a multi-layer perceptron module from input to output, and a normalization layer is set before each multi-head self-attention module and multi-layer perceptron module; each normalization layer is connected through a residual connection; The setting method of the multi-head self-attention module is: the multi-head self-attention module includes a window multi-head self-attention module and a shift window multi-head self-attention module; in several consecutive shift window transformation modules, from input to output, the multi-head self-attention module in the odd-numbered shift window transformation module is: the window multi-head self-attention module, and the multi-head self-attention module in the even-numbered shift window transformation module is: the shift window multi-head self-attention module.

5. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: The discriminator 1 passes through the first convolution module, the second convolution module, the third convolution module and the fully connected layer from input to output; the second convolution module and the third convolution module both use batch normalization layers; the first convolution module, the second convolution module and the third convolution module all use LeakyReLU activation functions; the fully connected layer uses tanh activation functions; the structure of the discriminator 2 is consistent with that of the discriminator 1.

6. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: The total loss of the improved generator module includes adversarial loss and content loss; the total loss calculation formula of the improved generator module is: ,in Represents resistance to loss, represents the training parameters, Indicates content loss.

7. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: During iterative training, the generator network learning rate of the improved PFNET model was set to 0.0001, the optimizer was AdamW, the dual discriminator network learning rate was 0.00001, the optimizer was Adam, the random rotation angle of the training set image was 5 degrees, the probability of random horizontal flipping was 5, and the probability of random vertical flipping was 5.

8. The color polarization image fusion method based on PFNET improvement according to claim 1 is characterized in that: When the performance test index reaches a fixed threshold, it is specifically: Information entropy: 7.0, Standard Deviation: , Visual Information Fidelity: , Structural Similarity Index: And the mutual information: .

9. The color polarization image fusion method based on PFNET improvement according to claim 1, characterized in that: The implementation process of inputting the verification data set into the executable PFNET improved model to obtain the RGB fused image includes the following steps: S91, the linear polarization image DOLP and the intensity image in the training set Convert from RGB color space to YCrCb color space to obtain brightness information and Cr and Cb channel information; S92, the uplink encoder module and the downlink encoder module respectively extract features from the brightness information, and concatenate the feature information extracted by the uplink encoder module and the downlink encoder module to obtain brightness features; S93, the brightness feature is sequentially passed through a fusion module and a decoder module to obtain a reconstructed image; S94, inputting the reconstructed image and brightness information into discriminator 1 and discriminator 2 for adversarial training to obtain a feedback signal; S95 , splicing the reconstructed image with the Cr and Cb channel information, and converting the spliced ​​image into an RGB color space to obtain an RGB fused image.

10. The color polarization image fusion method based on PFNET improvement according to claim 9 is characterized in that: The feedback signal is used to optimize the improved generator module.

Images