Polarization image fusion method and system based on multi-scale feature fusion

Abstract

The invention relates to a polarization image fusion method and system based on multi-scale feature fusion. The method comprises the following steps: acquiring an original polarization image of a polarization camera; according to the original polarization image, a light intensity image, a linear polarization degree image and a linear polarization angle image are obtained through Stokes vector calculation; performing feature extraction and fusion on the light intensity image, the linear polarization degree image and the linear polarization angle image in a neural network to form a fused image, and training the neural network based on a polarization characteristic retention loss function and a multi-scale feature loss function; and inputting the original polarization image into the trained neural network to obtain a polarization image with polarization information reserved. According to the invention, image fusion can be carried out by adding one piece of polarization information, so that the image quality is improved, and the polarization information contained in the image is effectively reserved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision, and in particular to a polarization image fusion method and system based on multi-scale feature fusion. Background Art

[0002] With the continuous advancement of computer vision and image processing technology, image fusion technology has been widely used in many fields, such as remote sensing image processing, medical image analysis, and autonomous driving. In these applications, how to extract the most valuable information from images acquired from different sources or at different time points and effectively fuse them has become an important research topic. The image fusion method based on deep learning can automatically learn the fusion strategy of multi-source images and generate high-quality fused images by designing a specific neural network architecture and optimizing the loss function.

[0003] Traditional image fusion algorithms mainly rely on a variety of mathematical and signal processing techniques to obtain fused images by weighting and synthesizing information from images from different sources. These methods usually include: image fusion based on wavelet transform, which uses the frequency domain features of the image to process the image; image fusion based on region segmentation, which divides the image into regions and fuses the features of each region; and fusion algorithms based on optimization models, which optimize the effect of image fusion by setting an objective function. Image fusion methods based on deep learning use advanced neural network architectures to learn how to effectively combine information from multi-source images and retain more details and high-quality features. The basic principles of image fusion based on deep learning include: first, prepare a large number of multi-source image pairs as training data sets, which can be images from different sensors, different angles or different times; second, design suitable deep neural network structures, such as convolutional neural networks (CNNs), generative adversarial networks (GANs) or graph convolutional networks (GCNs), which can effectively extract and fuse local features and global information from different image sources; extract meaningful features from each image through the forward propagation process of the neural network, which include key elements such as edges, textures and colors; use the target image in the training data set to guide the network to learn how to effectively fuse multi-source images. This process is usually achieved by minimizing the difference between the fused image and the target image, which is quantified by the loss function; use optimization methods such as backpropagation algorithms and gradient descent to continuously adjust the network weights until the quality of the fused image is optimal; finally, use the test set to evaluate the performance of the network and output a high-quality fused image. The evaluation indicators can include image clarity, information retention and visual effects. In practical applications, especially in the fields of remote sensing and medical imaging, the optimization and efficiency of image fusion technology still face challenges, especially in large-scale data processing and real-time performance requirements. Summary of the invention

[0004] In order to solve the problem that the fused image quality of the traditional method is poor and the polarization information in the super-resolution image cannot be effectively retained, the present disclosure proposes a polarization image fusion method based on multi-scale feature fusion to solve the above problem.

[0005] According to one aspect of the present disclosure, a polarization image fusion method based on multi-scale feature fusion is provided, comprising: S10, obtaining an original polarization image from a polarization camera; S20, according to the original polarization image, using Stokes vector calculation to obtain a light intensity image, a linear polarization degree image, and a linear polarization angle image; S30, extracting features from the light intensity image, the linear polarization degree image, and the linear polarization angle image in a neural network and fusing them to form a fused image, and training the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; S40, inputting the original polarization image into the trained neural network to obtain a polarization image retaining polarization information.

[0006] Preferably, according to the original polarization image, the light intensity image, the linear polarization degree image and the linear polarization angle image are obtained by using Stokes vector calculation, which are expressed as: , , , in, , , In the formula, , , , The original polarization images are at four polarization angles: 0°, 45°, 90°, and 135°. is the intensity image covering all polarization angles, is the polarization difference image with polarization angles of 0° and 90°, is the polarization difference image with polarization angles of 45° and 135°, is the linear polarization image, is the linear polarization angle image.

[0007] Preferably, the neural network is trained based on a polarization characteristic preservation loss function and a multi-scale feature loss function, and the polarization characteristic preservation loss function is expressed as: , Where, L PFor polarization-preserving loss functions, The coefficient corresponding to the polarization-preserving loss function, L DOLP is the linear polarization loss, L AoLP is the linear polarization angle loss; The multi-scale feature loss function is expressed as: , In the formula, is the multi-scale feature loss function, Indicates that the window is w of, x and y The local structural similarity loss between x and y are two different images or feature maps, , is the weight coefficient, is the size of the window, is the feature map of light intensity in the image, is the characteristic graph of linear polarization degree in the image, is the feature map of the image regarding the linear polarization angle, and is the feature map after feature extraction in the image.

[0008] Preferably, the linear polarization degree loss is obtained by performing regional weighting on the linear polarization degree image, which is expressed as: , , In the formula, is the weight of the linear polarization image, and are the feature maps of the linear polarization images of the fused image and the real image, respectively. is the pixel point in the image ( x , y ), W is 0.6, N is the number included when summing over all feature points.

[0009] Preferably, the linear polarization angle loss is obtained by performing regional weighting on the linear polarization angle image, and is expressed as: , , In the formula, is the weight of the linear polarization angle image, and are the feature maps of the linear polarization angle images of the fused image and the real image, respectively. is the target polarization angle, is the pixel point in the image ( x, y ) at the polarization angle.

[0010] Preferably, extracting features from the light intensity image, the linear polarization degree image and the linear polarization angle image and fusing them to form a fused image in a neural network comprises: A dense network is used to extract features from the intensity image, the linear polarization degree image and the linear polarization angle image; the feature extraction results are spliced ​​through a splicing function; and the spliced ​​features are fused together to obtain a fused image.

[0011] Preferably, a dense network is used to extract features from the intensity image, the linear polarization degree image and the linear polarization angle image, which can be expressed as: , In the formula, H l Indicates l The nonlinear transformation function of the layer, [F0,F1,…,F l−1 ] represents the feature concatenation operation, and F0 means that the input feature map is the intensity image, the linear polarization degree image and the linear polarization angle image.

[0012] According to one aspect of the present disclosure, a polarization image fusion system based on multi-scale feature fusion is provided, comprising: An original polarization image acquisition module is used to acquire the original polarization image of the polarization camera; The Stokes vector calculation module uses the Stokes vector calculation to obtain the light intensity image, the linear polarization degree image and the linear polarization angle image according to the original polarization image; A neural network training module is used to extract features of the light intensity image, the linear polarization degree image and the linear polarization angle image in the neural network and fuse them to form a fused image, and to train the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; The polarization image acquisition module inputs the original polarization image into the trained neural network to obtain a polarization image that retains polarization information.

[0013] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the above-mentioned polarization image fusion method based on multi-scale feature fusion.

[0014] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the polarization image fusion method based on multi-scale feature fusion is implemented.

[0015] Compared with the prior art, the beneficial effects of the present disclosure are: 1) The present invention uses a pre-trained deep neural network to extract complementary features of polarization images, and uses these features to generate fused images. It can automatically learn the potential associations and features between images without high-quality training samples, thereby achieving more accurate image fusion.

[0016] 2) This paper applies the generative adversarial network to the image fusion task, so that the model can extract and fuse the optimal features from different source images, thereby generating high-quality fused images.

[0017] 3) The present invention can significantly improve the texture quality of the fused image and effectively preserve the polarization information in the super-resolution image.

[0018] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0019] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0021] Figure 1 A flow chart of a polarization image fusion method based on multi-scale feature fusion is shown; Figure 2 A network schematic diagram showing polarization image fusion in Example 1 of the present disclosure is shown; Figure 3 The structural block diagram of the polarization image fusion system based on multi-scale feature fusion is shown. DETAILED DESCRIPTION

[0022] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0023] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0024] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0025] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0027] This embodiment proposes a polarization image fusion method based on multi-scale feature fusion. Figure 1 A flow chart of a polarization image fusion method based on multi-scale feature fusion is shown, the method comprising: S10, obtaining an original polarization image from a polarization camera; S20, according to the original polarization image, using Stokes vector calculation to obtain a light intensity image, a linear polarization degree image, and a linear polarization angle image; S30, extracting features from the light intensity image, the linear polarization degree image, and the linear polarization angle image in a neural network and fusing them to form a fused image, and training the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; S40, inputting the original polarization image into the trained neural network to obtain a polarization image retaining polarization information.

[0028] Based on the above steps, the disclosed embodiment uses a pre-trained deep neural network to extract complementary features of polarization images, and uses these features to generate fused images, which can automatically learn the potential associations and features between images without high-quality training samples, thereby achieving more accurate image fusion. The disclosed embodiment also applies a generative adversarial network to the image fusion task, so that the model can extract and fuse the optimal features from different source images, thereby generating a high-quality fused image.

[0029] In order to further illustrate the optional implementation methods of the present disclosure, the following description is made: S10. Acquire an original polarization image from a polarization camera.

[0030] In this embodiment, the original polarization image is acquired by a planar array polarization camera designed based on the principle of focal plane segmentation polarization imaging, and the original polarization image includes polarization state images at four angles, namely, polarization state images at 0°, 45°, 90° and 135°.

[0031] Among them, the polarization state image is mainly obtained by using a camera with a focal plane segmentation area array sensor (such as the IMX250MYR model). The principle of the polarization sensor in the polarization camera is to add a layer of polarizer above the photodiode based on the traditional CMOS image sensor. Four polarizers with different angles (90°, 45°, 135° and 0°) are placed on a single pixel, and each group of four pixels is used as a calculation unit. The degree and direction of polarization can be calculated by the association between polarizers in different directions. By performing relevant operations on the image directly obtained by the polarization camera, the steps of segmentation, reorganization and recalculation are omitted, which reduces the complexity of system use and improves the running speed of the overall process.

[0032] S20. According to the original polarization image, a light intensity image, a linear polarization degree image and a linear polarization angle image are obtained by using Stokes vector calculation.

[0033] In this embodiment, according to the original polarization image, the light intensity image, the linear polarization degree image and the linear polarization angle image are obtained by using the Stokes vector calculation, which are expressed as: , , , in, , , In the formula, , , , The original polarization images are at four polarization angles: 0°, 45°, 90°, and 135°. is the intensity image covering all polarization angles, is the polarization difference image with polarization angles of 0° and 90°, is the polarization difference image with polarization angles of 45° and 135°, is the linear polarization image, is the linear polarization angle image.

[0034] S30, extracting features of the light intensity image, the linear polarization degree image and the linear polarization angle image in a neural network and fusing them to form a fused image, and training the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function.

[0035] In this embodiment, in the generator of the dense generative adversarial network (D-GAN), a dense network (DenseNet) is used to extract features from the obtained intensity image (S0), degree of linear polarization image (DoLP) and angle of linear polarization image (AoLP) and fuse them to obtain a fused image (Image (Fusion)), as shown in the attached figure. Figure 2 The framework of the dense generative adversarial network is shown in Figure 1, where Generator G is the generator and Discriminator D is the discriminator. The task of the generator is to generate samples (Tanh) similar to the real data after the fusion of the intensity image (S0), the degree of linear polarization image (DoLP) and the angle of linear polarization image (AoLP) from random noise, while the task of the discriminator is to distinguish these generated data from the real data. Through continuous confrontation and optimization, the two finally generate realistic data Image. Figure 2The Generator G in the figure is implemented by a dense network (DenseNet), wherein the dense network includes an initial convolution layer (Initial Convolution Layer), a convolutional neural network module (DenseBlock) and an upsampling layer (Upsampling Layer) connected in sequence, wherein the main function of the PixelShuffle (pixel reorganization) in the upper layer is to obtain a high-resolution feature map through convolution and multi-channel reorganization of the low-resolution feature map; the convolutional neural network module (Dense Block) is implemented through 6 layers of convolution (Conv) and splicing (Concat); and the discriminator (Discriminator D) is finally determined to be Fake / Real through convolution (Conv), data normalization module (Norm), LeakyReLU activation function and multi-channel convolution. Generative adversarial networks (GANs) can improve the fusion effect to a certain extent, especially in complex scenes. Although the training process of the generative adversarial network is relatively complex and may require a long training time, the quality of the image it generates is superior in multiple application fields. Therefore, this embodiment uses a generative adversarial network as a benchmark model.

[0036] This embodiment applies a generative adversarial network to the image fusion task, so that the model can extract and fuse the optimal features from different source images, thereby generating a high-quality fused image. The core idea of ​​this method is to train a generative network to learn adversarially with the discriminative network, so that the generative network can generate realistic fused images, while the discriminative network improves the performance of the generative network by comparing the differences between the real image and the generated image. In this process, the generative adversarial network can not only effectively handle the problem of detail fusion between images of different modalities, but also retain the global structure of the image and avoid common blurring and artifacts. By using a pre-trained generative adversarial network model, it is possible to automatically learn the potential associations and features between images without high-quality training samples, thereby achieving more accurate image fusion.

[0037] The neural network is trained based on the polarization characteristic preservation loss function and the multi-scale feature loss function, and the polarization characteristic preservation loss function is expressed as: , Where, L P For polarization-preserving loss functions, The coefficient corresponding to the polarization-preserving loss function, L DOLP is the linear polarization loss, L AoLP is the linear polarization angle loss; The multi-scale feature loss function is expressed as: , In the formula, is the multi-scale feature loss function, , is the weight coefficient, Indicates that the window is w of x and y The local structural similarity loss between x and y are two different images or feature maps, w For the size of the window, multiple windows w ∈{3, 5, 7, 9, 11}, is the feature map of light intensity in the image, is the characteristic graph of linear polarization degree in the image, is the characteristic map of the linear polarization angle in the image, It is the feature map after feature extraction in the image.

[0038] It is expressed as: , In the formula, For images x In the window w The area inside, for The mean of and They are The variance and , The covariance of , and The same meaning, C 1. C 2 is a constant.

[0039] in, Using multiple windows to extract features of different scales is more accurate than using only SSIM as the loss function. , Defined as: , , In the formula, g( x ) = max(x, 0.0001)is a correction function to increase the robustness of the solution, max(a, b)represents the larger value of a and b, The window corresponding to the light intensity w The correction function of the variance of The window corresponding to the linear polarization degreew The correction function of the variance of The linear polarization angle corresponds to the window w The correction function of the variance.

[0040] Furthermore, the polarization preservation characteristic loss function is obtained by weighted summing the polarization characteristic preservation loss function and the multi-scale feature loss function, which is expressed as: , In the formula, is the coefficient corresponding to the loss function.

[0041] The linear polarization loss is obtained by performing regional weighting on the linear polarization image, and is expressed as: , , In the formula, is the weight of the linear polarization image, and are the feature maps of the linear polarization images of the fused image and the real image, respectively. is the pixel point in the image ( x , y ), W is 0.6, N is the number included when summing over all feature points.

[0042] The linear polarization angle loss is obtained by performing regional weighting on the linear polarization angle image, and is expressed as: , , In the formula, is the weight of the linear polarization angle image, and are the feature maps of the linear polarization angle images of the fused image and the real image, respectively. is the target polarization angle, is the pixel point in the image ( x , y ) at the polarization angle.

[0043] Furthermore, a dense network is used to extract features from the obtained intensity image, linear polarization degree image, and linear polarization angle image, which can be expressed as: , In the formula, H l Indicates l The nonlinear transformation function of the layer, [F0,F1,…,F l−1 ] represents the feature concatenation operation, and F0 means that the input feature map is the intensity image, the linear polarization degree image and the linear polarization angle image.

[0044] Among them, H l Contains the following operations: , In the formula, Conv represents the convolution operation, BN represents batch normalization, and P ReLU is the activation function.

[0045] The feature fusion process is as follows: The feature maps of the intensity image (S0), degree of linear polarization image (DoLP) and angle of linear polarization image (AoLP) are concatenated along the channel dimension to form a new feature map, which is expressed as: , In the formula, is the new feature map formed after splicing. is the feature map of the intensity image (S0), is the characteristic map of the degree of linear polarization image (DoLP), It is the feature map of the Angle of Linear Polarization (AoLP) image; Adding the polarization attention mechanism can make full use of polarization information. Assuming the input feature map is , whose attention weight is set to It is expressed as: , In the formula, and is the weight matrix used to calculate the query and key-value vectors, and is the input feature map at position i and j The eigenvector at , ⟨⋅,⋅> represents the inner product operation, For the i Position to j The attention weight of each position; Final Output is the feature map weighted by the attention weight, expressed as: .

[0046] The process uses a pre-trained network to extract deep features of the intensity image (S0), degree of linear polarization image (DoLP) and angle of linear polarization image (AoLP); stitches these features together through a splicing function; uses the spliced ​​features to fuse the images, usually through an upsampling layer (such as Pixelshuffle); generates adversarial optimization, and during the fusion process, it optimizes the image through generative adversarial multiple times to make it closer to the real fused image.

[0047] S40, inputting the original polarization image into the trained neural network to obtain a polarization image retaining polarization information.

[0048] In this embodiment, the network diagram of polarization image fusion is as follows: Figure 2 As shown in the figure, the original polarization image is calculated by Stokes vector to generate the intensity image (S0), the degree of linear polarization image (DoLP) and the angle of linear polarization image (AoLP). The feature maps of the intensity image (S0), the degree of linear polarization image (DoLP) and the angle of linear polarization image (AoLP) are spliced ​​along the channel dimension to form a new feature map. The fused image obtained by fusion in the generator is used as the input of the network, and the source image is used as the output. The neural network is preliminarily iteratively trained to obtain a network parameter that can restore image information and polarization information; the preliminarily trained neural network is used to take the original polarization image as input and output a polarization image that retains the polarization information.

[0049] This embodiment is based on a generative adversarial network, which enables the model to extract multi-source features from multi-scale polarization images and fuse them together without supervision, without the need for any training samples of fused images. The key idea of ​​this method is to use a pre-trained deep neural network to extract complementary features of polarization images, and use these features to generate and fuse the corresponding fused images, which can significantly improve the texture quality of the fused image and effectively preserve the polarization information in the image.

[0050] As another aspect of the embodiments of the present disclosure, a polarization image fusion system 100 based on multi-scale feature fusion is also provided. Figure 3 As shown, including: The original polarization image acquisition module 1 acquires the original polarization image of the polarization camera; Stokes vector calculation module 2, based on the original polarization image, obtains the light intensity image, the linear polarization degree image and the linear polarization angle image by using Stokes vector calculation; A neural network training module 3 is used to extract features of the light intensity image, the linear polarization degree image and the linear polarization angle image in the neural network and fuse them to form a fused image, and to train the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; The polarization image acquisition module 4 inputs the original polarization image into the trained neural network to obtain a polarization image that retains polarization information.

[0051] In the absence of any contradiction, the above modules in the system of the embodiment of the present disclosure can implement any implementation of the above method.

[0052] Based on the description of the above embodiments, it can be seen that the embodiments of the present disclosure can achieve the following technical effects: 1) The disclosed embodiment uses a pre-trained deep neural network to extract complementary features of polarization images, and uses these features to generate fused images. It can automatically learn the potential associations and features between images without high-quality training samples, thereby achieving more accurate image fusion.

[0053] 2) The disclosed embodiment applies a generative adversarial network to the image fusion task, so that the model can extract and fuse the optimal features from different source images, thereby generating a high-quality fused image.

[0054] 3) The disclosed embodiments can significantly improve the texture quality of the fused image and effectively preserve the polarization information in the super-resolution image.

[0055] The disclosed embodiment also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the polarization image fusion method based on multi-scale feature fusion. The electronic device can be provided as a terminal, a server, or other forms of devices.

[0056] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the polarization image fusion method based on multi-scale feature fusion is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0057] Those skilled in the art can understand that in the above-mentioned polarization image fusion method and system based on multi-scale feature fusion in the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0058] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0059] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A polarization image fusion method based on multi-scale feature fusion, characterized in that: The steps include: S10, obtaining an original polarization image from a polarization camera; S20, according to the original polarization image, using Stokes vector calculation to obtain a light intensity image, a linear polarization degree image, and a linear polarization angle image; S30, extracting features from the light intensity image, the linear polarization degree image, and the linear polarization angle image in a neural network and fusing them to form a fused image, and training the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; S40, inputting the original polarization image into the trained neural network to obtain a polarization image retaining polarization information.

2. The method according to claim 1, characterized in that: According to the original polarization image, the intensity image, linear polarization degree image and linear polarization angle image are calculated using the Stokes vector, which are expressed as: ， ， ， in, ， ， In the formula, , , , The original polarization images are at four polarization angles of 0°, 45°, 90° and 135°. is the intensity image covering all polarization angles, is the polarization difference image with polarization angles of 0° and 90°, is the polarization difference image with polarization angles of 45° and 135°, is the linear polarization image, is the linear polarization angle image.

3. The method according to claim 1, characterized in that The neural network is trained based on the polarization characteristic preservation loss function and the multi-scale feature loss function, and the polarization characteristic preservation loss function is expressed as: ， Where, L P For polarization-preserving loss functions, The coefficient corresponding to the loss function for polarization preservation, L DOLP is the linear polarization loss, L AoLP is the linear polarization angle loss; The multi-scale feature loss function is expressed as: ， In the formula, is the multi-scale feature loss function, Indicates that the window is w of x and y The local structural similarity loss between x and y are two different feature maps. , is the weight coefficient, w is the size of the window, is the feature map of light intensity in the image, is the characteristic graph of linear polarization degree in the image, is the characteristic map of the linear polarization angle in the image, It is the feature map after feature extraction in the image.

4. The method according to claim 3, characterized in that: The linear polarization loss is obtained by performing regional weighting on the linear polarization image, and is expressed as: ， ， In the formula, is the weight of the linear polarization image, and are the feature maps of the linear polarization images of the fused image and the real image, respectively. is the pixel point in the image ( x , y ), W is 0.6, N is the number included when summing over all feature points.

5. The method according to claim 3, characterized in that: The linear polarization angle loss is obtained by performing regional weighting on the linear polarization angle image, and is expressed as: ， ， In the formula, is the weight of the linear polarization angle image, and are the feature maps of the linear polarization angle images of the fused image and the real image, respectively. is the target polarization angle, is the pixel point in the image ( x , y ) at the polarization angle.

6. The method according to claim 5, characterized in that Extracting features of the light intensity image, the linear polarization degree image and the linear polarization angle image in a neural network and fusing them to form a fused image include: A dense network is used to extract features from the intensity image, the linear polarization degree image and the linear polarization angle image; the feature extraction results are spliced ​​through a splicing function; and the spliced ​​features are fused together to obtain a fused image.

7. The method according to claim 1, characterized in that A dense network is used to extract features from the intensity image, linear polarization image, and linear polarization angle image, which can be expressed as: ， In the formula, H l Indicates l The nonlinear transformation function of the layer, [F0,F1,…,F l−1 ] represents the feature concatenation operation, and F0 means that the input feature map is the intensity image, the linear polarization degree image and the linear polarization angle image.

8. Polarization image fusion system based on multi-scale feature fusion, characterized in that: include: An original polarization image acquisition module is used to acquire the original polarization image of the polarization camera; The Stokes vector calculation module uses the Stokes vector calculation to obtain the light intensity image, the linear polarization degree image and the linear polarization angle image according to the original polarization image; A neural network training module is used to extract features of the light intensity image, the linear polarization degree image and the linear polarization angle image in the neural network and fuse them to form a fused image, and to train the neural network based on a polarization characteristic preservation loss function and a multi-scale feature loss function; The polarization image acquisition module inputs the original polarization image into the trained neural network to obtain a polarization image that retains polarization information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the polarization image fusion method based on multi-scale feature fusion described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the polarization image fusion method based on multi-scale feature fusion described in any one of claims 1 to 7 is implemented.

Images