Polarimetric image fusion system and method

Abstract

The present application relates to the technical field of image processing, and particularly relates to a polarized image fusion system and method, which comprises an encoder, a fusion layer and a decoder. The encoder adopts a double-branch feature extraction method cooperating gradient residual dense and SwinTransformer to extract local polarization features and global intensity information of intensity images and linear polarization degree images. In the fusion layer, an affine coupling layer is used to establish the correlation between the two types of features, to enhance the polarization features without loss and to perform fusion. In the decoder, a multi-dimensional convolution attention mechanism and a dynamic feedforward are cooperated to decode the fused features step by step, to recover and retain the high-frequency features and scene details in the fused features, so as to improve the image definition and obtain the fused image. In the method provided in the present application, a semantic loss function is used to guide the high-level semantic information to flow back to the fusion system, to guide the training of the fusion system, and to enhance the high-level visual task application properties of the fused image.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and particularly relates to a polarized image fusion system and method. BACKGROUND

[0002] Quality and target recognition ability. The intensity image reflects the intensity details and brightness information of the scene, and can accurately reflect the appearance of the target object, but the description of the surface characteristics and reflection information is very limited. The linear polarization degree image reveals the surface roughness, material characteristics and other details by capturing the reflected polarized light information of the object surface, and has strong perception for transparent and reflective target objects. Therefore, the fusion of the intensity image and the linear polarization degree image can obtain brightness information and object reflection characteristics at the same time, especially in complex road scenes and harsh weather environments, which can enhance the target detection and object recognition accuracy.

[0003] At present, image fusion methods are mainly divided into two categories: traditional methods and deep learning. The mainstream traditional methods include fusion methods based on multi-scale analysis and sparse representation learning. The former has classic Laplacian Pyramid (LP), Wavelet Transform (WT), etc.; the latter mainly has joint sparse representation JSR, convolutional sparse representation CSR, etc. With the continuous development of deep learning and according to the structure characteristics of neural networks, the fusion methods based on deep learning mainly include convolutional neural network (CNN), autoencoder network (AE) and generative adversarial network (GAN).

[0004] However, the existing image fusion methods still tend to pursue higher image objective evaluation indicators, ignoring the visual effects presented by the fused image itself and the application performance in subsequent high-level visual tasks such as semantic segmentation or target detection, resulting in problems such as missing important features, blurred texture and poor segmentation results in the fused image. SUMMARY

[0005] Therefore, the present application aims to provide a polarized image fusion system and method, which captures local polarization details such as edges and textures through gradient dense residual by using an encoder with double-branch feature extraction, captures global intensity information by using the moving window mechanism and hierarchical structure of SwinTransformer; in the fusion layer, two types of features with complementary effects are enhanced and fused by using an affine coupling layer; in the decoder, multi-scale reconstruction fusion features are obtained by using a multi-dimensional convolution attention mechanism and a dynamic feedforward mechanism.

[0006] To achieve the above purpose, the technical scheme of the present application is as follows:

[0007] A polarized image fusion system comprises: an encoder comprising an intensity encoding branch and a polarization encoding branch; the intensity encoding branch performs feature extraction based on a Swin Transformer and gradient extraction on an input intensity image to obtain intensity encoding features and intensity gradient features, respectively; the polarization encoding branch performs feature extraction based on a Swin Transformer and gradient extraction on an input linear polarization degree image to obtain polarization encoding features and polarization gradient features, respectively; the two encoding features and the two gradient features are added correspondingly to obtain encoding features and gradient features containing the linear polarization degree image and the intensity image simultaneously; a fusion device fuses the encoding features and the gradient features output by the dual-branch encoder using an affine coupling layer to obtain fusion features; and a decoder decodes the fusion features step by step using a multi-dimensional convolution attention mechanism and a dynamic feedforward mechanism to obtain a fusion image of the intensity image and the linear polarization degree image.

[0008] Further, the intensity encoding branch and the polarization encoding branch each comprise: a shallow feature extraction module that performs shallow feature extraction on an input image; a Swin Transformer module that performs a moving window attention operation on the shallow features to obtain corresponding intensity features; and a gradient extraction module that performs a dense residual operation on the shallow features and simultaneously calculates gradient information of the shallow features using a gradient operator, and combines the gradient information with the output features of the residual operation to obtain corresponding gradient features.

[0009] Further, the fusion device comprises a plurality of affine coupling layers connected in series; in each affine coupling layer, the input encoding features and gradient features are bidirectionally mapped using a mapping function; and the encoding features and gradient features output by the last affine coupling layer are integrated to obtain coupling features.

[0010] Further, each affine coupling layer comprises a first mapping function and a second mapping function, and the first mapping function and the second mapping function each perform multiple continuous convolution operations on the input features; along a data processing direction, the number of channels of the convolution performed by the first mapping function gradually increases, and the number of channels of the convolution performed by the second mapping function gradually decreases; the forward mapping in each affine coupling layer is as follows:

[0011] ;

[0012] ;

[0013] wherein, represents the gradient features output by the i-th affine coupling layer, represents the encoding features output by the i-th affine coupling layer, MF1 represents the first mapping function, MF2 represents the second mapping function, and exp represents an exponential function; the reverse mapping in each affine coupling layer is as follows:

[0014] ;

[0015] .

[0016] Further, the decoder comprises a plurality of decoding modules in succession, each decoding module comprising a feature extraction part and a hierarchical decoding part, and each of the feature extraction part and the hierarchical decoding part comprises a plurality of Transformer blocks, each Transformer block sequentially performing a multi-dimensional convolution attention operation and a dynamic feedforward operation on the input feature; in the feature extraction part, the input feature is extracted by the plurality of Transformer blocks in a hierarchical manner, and the feature extracted at each level is input into the hierarchical decoding part; in the hierarchical decoding part, the smallest scale feature output by the feature extraction part is decoded in a hierarchical manner in combination with the feature of the corresponding scale from the feature extraction part; the input feature and the output feature of the feature extraction part are combined through a skip connection to obtain a fused image.

[0017] Further, the process of the multi-dimensional convolution attention operation comprises: extracting a query matrix, a key matrix and a value matrix from the input feature, respectively performing a matrix reshaping operation on the query matrix and the key matrix, combining the two features after the Softmax operation to obtain a transposed attention map; combining the transposed attention map with the value matrix, and then pixel-by-pixel summing with the input feature to obtain an output feature.

[0018] Further, the process of the dynamic feedforward operation comprises: performing a double-line parallel convolution operation on the input feature, performing a GELU activation operation on the feature after one of the convolution operations to obtain a gating weight, performing a dot product operation with the feature after the other convolution operation, and then pixel-by-pixel adding with the input feature to obtain an output feature.

[0019] A polarization image fusion method, comprising:

[0020] S1: obtaining a data set containing a linear polarization degree image, a corresponding intensity image, and a real fusion image of the two, and making semantic segmentation labels required for segmentation tasks to obtain a training set;

[0021] S2: constructing a polarization image fusion system as provided by the present application, and training the polarization image fusion system using the training set obtained in step S1 to obtain a polarization image fusion model; in the training process, the content loss and semantic loss between the predicted fusion image output by the polarization image fusion model and the corresponding real fusion image are used to train the polarization image fusion model;

[0022] S3: inputting the linear polarization degree image to be fused and its corresponding intensity image into the polarization image fusion model trained in step S2 to obtain a final fusion image.

[0023] Further, the polarization image fusion model is trained in step S2 by a composite loss function of the following formula:

[0024] L=L content +β(m-1)×L semantic ;

[0025] Wherein, L represents the composite loss function, L content represents the content loss, L semantic represents the semantic loss, β represents the balance parameter of the semantic loss and the content loss, and m represents the iteration number of training; the content loss is obtained by the following formula:

[0026] ;

[0027] Wherein, H and W respectively represent the height and width of the fusion image, I F represents the predicted fusion image, S0 represents the intensity image, DoLP represents the linear polarization degree image, represents the gradient operator, and α represents the content weight; the calculation process of the semantic loss includes: inputting the real fusion image and the predicted fusion image into the trained real-time semantic segmentation network to obtain respective semantic segmentation results, and calculating the semantic loss by the following formula:

[0028] ;

[0029] Wherein, h, w and c respectively represent the index of the image height direction, the width direction and the channel direction, L S0 represents a binary vector transformed from the segmentation label L s of the real fusion image, I s represents the semantic segmentation result corresponding to the predicted fusion image, I sa represents the auxiliary segmentation result corresponding to the predicted fusion image, and λ represents the semantic loss weight.

[0030] Compared with the prior art, the application can achieve the following beneficial effects:

[0031] In the polarized image fusion system and method, the encoder adopts a double-branch feature extraction mode, gradient dense residual is used to capture local polarized details such as edges and textures, and the moving window mechanism and hierarchical structure of SwinTransformer are used to capture global intensity information; in the fusion layer, the affine coupling layer is used to enhance and fuse the two types of features that show complementary effects; in the decoder, the multi-dimensional convolution attention mechanism and the dynamic feedforward mechanism are used in cooperation to perform multi-scale reconstruction and fusion of the features, and the fused image is recovered layer by layer; in addition, in the polarized image fusion method provided by the application, the model is trained by combining content loss and semantic loss, so that the fused image can contain stable intensity and polarization information, and has good application performance in subsequent semantic segmentation visual tasks. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of the present application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0033] Figure 1 A schematic diagram of the polarized image fusion system according to an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the Swin Transformer module according to an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the SwinTransformer layer according to an embodiment of the present application;

[0036] Figure 4 A schematic diagram of the gradient extraction module according to an embodiment of the present application;

[0037] Figure 5 A schematic diagram of the affine coupling layer according to an embodiment of the present application;

[0038] Figure 5 (a) A process diagram of forward mapping in the affine coupling layer according to an embodiment of the present application;

[0039] Figure 5 (b) A process diagram of reverse mapping in the affine coupling layer according to an embodiment of the present application;

[0040] Figure 6 A schematic diagram of the decoding module according to an embodiment of the present application;

[0041] Figure 7 A process diagram of the multi-dimensional convolution attention operation according to an embodiment of the present application;

[0042] Figure 8 Process schematic diagram of the dynamic feedforward operation described in the embodiments of the present invention;

[0043] Figure 9 Flowchart of the polarized image fusion method described in the embodiments of the present invention;

[0044] Figure 10 Training overall flow block diagram of the polarized image fusion method described in the embodiments of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in combination with the drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and do not constitute a limitation on the present invention.

[0046] It should be noted that the embodiments in the present invention and the features in the embodiments can be combined with each other without conflict.

[0047] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present invention and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second" and the like are only used for description purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features limited by "first", "second" and the like can explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0048] The present invention will be described in detail below with reference to the drawings and in combination with embodiments.

[0049] As Figure 1As shown, the polarization image fusion system described in the embodiments of the present application includes an encoder, a fusioner and a decoder. The encoder adopts the form of a double-branch feature extraction method, including an intensity encoding branch and a polarization encoding branch. The intensity encoding branch performs feature extraction based on Swin Transformer and gradient extraction on the input intensity image, respectively obtaining intensity encoding features and intensity gradient features. The polarization encoding branch performs feature extraction based on Swin Transformer and gradient extraction on the input linear polarization degree image, respectively obtaining polarization encoding features and polarization gradient features. The two encoding features and the two gradient features are added correspondingly to obtain encoding features and gradient features containing linear polarization degree images and intensity images at the same time. The fusioner fuses the encoding features and gradient features output by the double-branch encoder using an affine coupling layer to obtain fusion features. The decoder adopts a multi-dimensional convolution attention mechanism and a dynamic feedforward mechanism to decode the fusion features step by step to obtain the fusion image of the intensity image and the linear polarization degree image.

[0050] In some embodiments, the intensity encoding branch and the polarization encoding branch each include a shallow feature extraction module, a Swin Transformer module and a gradient extraction module. The shallow feature extraction module performs shallow feature extraction on the input image. The Swin Transformer module performs a moving window attention operation on the shallow features to obtain corresponding intensity features. The gradient extraction module performs a dense residual operation on the shallow features, and simultaneously calculates gradient information of the shallow features using a gradient operator, combines the gradient information with the output features of the residual operation, and obtains corresponding gradient features.

[0051] In the embodiments of the present application, the shallow feature extraction module is composed of a 3x3 convolution layer and an LReLU activation layer, and extracts 16-channel shallow features. The Swin Transformer module is as shown in Figure 2 It is composed of a block partition layer, a linear embedding layer, a block splicing layer and a Swin Transformer layer, and has a hierarchical structure divided into four consecutive stages. It extracts features using a moving window mechanism. The shallow feature map is calculated by 4 stages of W-MSA and SW-MSA alternately, each stage is responsible for extracting different levels of features, and captures global intensity information from fine granularity to coarse granularity, so that the Swin Transformer module gradually realizes multi-scale and cross-window feature extraction.

[0052] The shallow features are first divided into a series of overlapping block features by the block partition layer.

[0053] In the first stage, the block features are encoded by a linear embedding layer to convert the image block-level pixels into embedding vectors, and then enter two consecutive Swin Transformer layers, and the output features of the second Swin Transformer layer enter the second stage. As shown in Figure 3

[0054] ;

[0055] ;

[0056] ;

[0057] ;

[0058] wherein, F in represents the input features of the Swin Transformer layer, F out represents the output features of the Swin Transformer layer, F1~F3 represent the intermediate features in the Swin Transformer layer, LN represents layer normalization, and FFN represents feedforward network calculation.

[0059] In the second stage, the input features enter the block splicing layer for adjacent block merging, resolution reduction and channel number increase, and then enter six consecutive Swin Transformer layers.

[0060] The third stage and the fourth stage are similar to the second stage, and the difference lies in that there are four consecutive Swin Transformer layers in the third stage, and there are two consecutive Swin Transformer layers in the fourth stage.

[0061] In the embodiment of the present application, the gradient extraction module is as shown in Figure 4 The main flow is a dense residual operation, wherein a convolution combination composed of 3x3, 1x1 convolution and LReLU activation function is used, and a Sobel gradient operator and a 1x1 convolution layer are used to obtain gradient information. Finally, the output features of the dense residual operation and the gradient information are integrated through element-wise addition to extract local polarization features and granularity details, and the output gradient features are obtained.

[0062] ​In some embodiments, the fusioner comprises a plurality of affine coupling layers in cascade. The affine coupling layer adopts the affine coupling layer generalized by the three Real NVP models in Invertible Neural Networks (INN), and the input and output in the layer are bidirectionally mapped, and the complementary intensity and polarization features from the encoder are enhanced and coupled through the mapping function. Specifically, as shown in Figure 5 the encoding features and gradient features of the input are bidirectionally mapped by using the mapping function; and the encoding features and gradient features output by the last affine coupling layer are integrated to obtain the coupled features.

[0063] Furthermore, the first mapping function and the second mapping function are included in each affine coupling layer, and the first mapping function and the second mapping function both perform multiple continuous convolution operations on the input features. In the data processing direction, the number of channels of the convolution performed by the first mapping function gradually increases, and the number of channels of the convolution performed by the second mapping function gradually decreases.

[0064] The forward mapping in each affine coupling layer is shown in (a) of Figure 5 , and is specifically as follows:

[0065] ;

[0066] ;

[0067] wherein, represents the gradient feature output by the i-th affine coupling layer, represents the encoding feature output by the i-th affine coupling layer, MF1 represents the first mapping function, MF2 represents the second mapping function, and exp represents the exponential function;

[0068] The reverse mapping in each affine coupling layer is shown in (b) of Figure 5 , and is specifically as follows:

[0069] ;

[0070] .

[0071] The embodiment of the present application specifically adopts a 3-layer affine coupling layer, the data processing process of the first mapping function MF1 includes performing 3*3 convolution on the input feature for two times in succession, and performing batch normalization operation on the feature after each convolution, performing LReLU activation operation on the feature after the second batch normalization to obtain the output feature, and the channel number of the feature after each convolution becomes 2 times of the channel number before the convolution, the data processing process of the second mapping function MF2 includes performing 3*3 convolution on the input feature for two times in succession, and performing batch normalization operation on the feature after each convolution, performing LReLU activation operation on the feature after the second batch normalization to obtain the output feature, and the channel number of the feature after each convolution becomes 1 / 2 times of the channel number before the convolution. In the three-layer affine coupling layer, the features are coupled in an alternating manner of the first mapping function MF1 and the second mapping function MF2, and on the basis of ensuring that the feature information is lossless, the local polarization details and the global intensity features are integrated.

[0072] In some embodiments, the decoder includes a plurality of decoding modules in succession, each decoding module including a feature extraction part and a step-by-step decoding part, and the feature extraction part and the step-by-step decoding part each include a plurality of Transformer blocks, each Transformer block sequentially performing a multi-dimensional convolution attention operation and a dynamic feedforward operation on the input feature, the multi-dimensional convolution attention operation realizing local and global pixel interaction aggregation by calculating cross-channel covariance, and the dynamic feedforward operation controlling information flow of each level in the channel through a gating mechanism, allowing each level to focus on complementary details of other levels. In the feature extraction part, a plurality of Transformer blocks are used to extract features of the input feature step by step, and the features extracted at each step are input into the step-by-step decoding part. In the step-by-step decoding part, the smallest scale feature output by the feature extraction part is decoded step by step in combination with the features from the corresponding scale in the feature extraction part. Finally, the input feature and the output feature of the feature extraction part are combined through a skip connection to obtain a fused image.

[0073] In the present application, the decoder includes two decoding modules in succession, each decoding module as Figure 6As shown in the figure, the feature extraction part includes 4 Transformer blocks, and the step-by-step decoding part includes 4 Transformer blocks. Specifically, in the feature extraction part: the input feature enters the first Transformer block after 3x3 convolution, the output feature of the first Transformer block enters the second Transformer block and the step-by-step decoding part, the output feature of the second Transformer block enters the third Transformer block and the step-by-step decoding part, the output feature of the third Transformer block enters the fourth Transformer block and the step-by-step decoding part, and the output feature of the fourth Transformer block is spliced with the output feature of the third Transformer block and then enters the step-by-step decoding part; in the step-by-step decoding part: the spliced output feature enters the fifth Transformer block after 1x1 convolution, the output feature of the fifth Transformer block is spliced with the output feature of the second Transformer block, and then 1x1 convolution is performed to enter the sixth Transformer block, the output feature of the sixth Transformer block is spliced with the output feature of the first Transformer block and then enters the seventh Transformer block, the output feature of the seventh Transformer block directly enters the eighth Transformer block, and the output feature of the eighth Transformer block is added with the corresponding elements of the input feature after 3x3 convolution to obtain the fusion image.

[0074] In some embodiments, the process of the multi-dimensional convolution attention operation includes: extracting a query matrix, a key matrix and a value matrix from the input feature, respectively performing a matrix reshaping operation on the query matrix and the key matrix, combining the two features after the operation, and then performing a Softmax operation to obtain a transposed attention map; combining the transposed attention map with the value matrix, and then pixel-by-pixel summing with the input feature to obtain the output feature. In the embodiment of the application, the process of the multi-dimensional convolution attention operation is as shown in the figure Figure 7 As shown in the figure, the input feature with a size of HxWxC is normalized by layer, 1x1 and 3x3 depth separable convolution operations of 3 branches are performed to obtain a query matrix Q, a key matrix K and a value matrix V, so as to enhance local information, and H, W and C represent the height, width and channel number of the feature respectively. Then, the query matrix Q and the key matrix K are respectively subjected to a reshaping operation to change their sizes to HWxC, and then matrix multiplication is performed to generate a transposed attention map with a size of CxC, and matrix operation is performed with the value matrix V, and the result is restored to an HxWxC feature map through reshaping. Finally, 1x1 convolution is performed and pixel-by-pixel summation is performed with the input feature after layer normalization to obtain the output feature.

[0075] In some embodiments, the process of the dynamic feedforward operation includes: performing double-line parallel convolution operation on the input features, performing nonlinear activation on the features after one of the convolution operations by using GELU activation operation to obtain gating weights, performing dot product operation with the features after the other convolution operation, and then adding the input features pixel by pixel to obtain the output features.

[0076] In the embodiments of the present application, the process of the dynamic feedforward operation is as shown in Figure 8 Two branch 1x1 and 3x3 deep convolution operations are performed on the layer-normalized input feature map, one branch result is activated by GELU nonlinear activation to obtain weights, dot product operation is performed with the output feature map of the other branch, and the result is added to the input feature map pixel by pixel after 1x1 convolution layer to obtain the output features.

[0077] The present application also provides a polarized image fusion method, which combines Figure 9 and Figure 10 , comprising:

[0078] S1: Obtain a data set containing a linear polarization degree image, a corresponding intensity image, and a real fusion image of the two, and make semantic segmentation labels required for segmentation tasks for the data set to obtain a training set, so that it is adapted to the training process of the present application. In the embodiments of the present application, the public data set Rachel Blin is specifically selected.

[0079] S2: Construct the polarized image fusion system provided by the present application, and train the polarized image fusion system using the training set obtained in step S1 to obtain a polarized image fusion model.

[0080] In order to improve and enhance the visual quality and semantic information of the fusion image, the present application trains the fusion network using a joint interactive training strategy, and designs a joint loss function including content loss and semantic loss, that is, the content loss and semantic loss between the predicted fusion image output by the polarized image fusion model and the corresponding real fusion image are used to train the polarized image fusion model.

[0081] In some embodiments, the polarized image fusion model is trained by a composite loss function as follows in step S2:

[0082] L=L content +β(m-1)×L semantic ;

[0083] Wherein, L represents the composite loss function, L content represents the content loss, and L semanticrepresents semantic loss, β represents a balance parameter of semantic loss and content loss, and m represents the number of iterations of training. In the embodiment of the present application, the learning rate and the weight decay rate of the fusion network training process are updated by using the Adam optimizer, and in the embodiment of the present application, the learning rate is specifically 0.001, the weight decay rate is specifically 0.0002, the balance parameter β is preferably set to 0.1, and with the increase of the number of iterations m, the semantic loss will more accurately optimize the fusion network parameters.

[0084] In order to make the fusion network retain more intensity and polarization information and improve the subjective and objective evaluation of the image, the content loss composed of intensity loss and texture loss is designed in the present application, and the content loss is specifically as follows:

[0085] ;

[0086] Wherein, H and W respectively represent the height and width of the fusion image, I F represents the predicted fusion image, S0 represents the intensity image, DoLP represents the linear polarization degree image, represents the gradient operator, and α represents the content weight. In the content loss, is used to measure the intensity difference between the predicted fusion image and the corresponding real fusion image pixel, is used to measure the degree of retaining the details and polarization characteristics in the real fusion image by the predicted fusion image. The present application adjusts the pixel brightness consistency of the predicted fusion image and the corresponding real fusion image by the maximum selection strategy (i.e. the max function) and the norm operation, and introduces the gradient operator to integrate the texture information of the predicted fusion image and the real fusion image, so as to retain the fine polarization information and texture structure in the image. In the embodiment of the present application, the content weight α is preferably set to 10, and the gradient operator The Sobel gradient operator is preferably used.

[0087] In order to enhance the semantic information of the fusion image, the real-time semantic segmentation network is introduced in the training, and the semantic loss is used to guide the fusion network training. Specifically, the calculation process of the semantic loss includes:

[0088] The real fusion image and the predicted fusion image are input into the trained real-time semantic segmentation network to obtain the respective semantic segmentation results, and the semantic loss is calculated by the following formula:

[0089] ;

[0090] Wherein, h, w and c respectively represent the index of the image height direction, the width direction and the channel direction, L S0 represents a binary vector transformed from the segmentation label L s of the real fusion image, I s represents the semantic segmentation result corresponding to the predicted fusion image, and Isa The auxiliary segmentation result corresponding to the prediction fusion image output by the BiSeNet is represented. In the embodiment of the present application, the BiSeNet trained is used for semantic segmentation, λ represents the semantic loss weight, and the embodiment of the present application is preferably set to 0.1. In the training process, the learning rate and the weight decay rate of the BiSeNet segmentation network are 0.01 and 0.0005, respectively.

[0091] S3: inputting the linear polarization degree image to be fused and the intensity image corresponding thereto into the polarization image fusion model trained in step S2 to obtain a final fusion image.

[0092] It should be understood that the various forms of flow shown above can be used to reorder, add or delete steps. For example, each step described in the present disclosure can be executed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which is not limited herein.

[0093] The above specific embodiments do not constitute a limitation on the scope of protection of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A polarization image fusion system, characterized in that, include: The encoder includes an intensity coding branch and a polarization coding branch: The intensity coding branch performs feature extraction based on Swing Transformer and gradient extraction on the input intensity image to obtain intensity coding features and intensity gradient features, respectively. The polarization coding branch performs feature extraction based on Swing Transformer and gradient extraction on the input linear polarization degree image to obtain polarization coding features and polarization gradient features, respectively. The two coding features and two gradient features are added together to obtain the coding features and gradient features that simultaneously contain the linear polarization degree image and the intensity image. The fusion unit uses an affine coupling layer to fuse the encoded features and gradient features output by the dual-branch encoder to obtain fused features. The decoder uses a combination of multidimensional convolutional attention mechanism and dynamic feedforward to decode the fused features step by step to obtain the fused image of intensity image and linear polarization degree image; The decoder consists of multiple consecutive decoding modules. Each decoding module includes a feature extraction part and a step-by-step decoding part. Both the feature extraction part and the step-by-step decoding part include multiple Transformer blocks. Each Transformer block performs multidimensional convolutional attention operations and dynamic feedforward operations on the input features in sequence. In the feature extraction part, multiple Transformer blocks are used to extract features from the input features step by step, and the features extracted at each level are input into the decoding part step by step. In the step-by-step decoding section, features of the corresponding scale from the feature extraction section are combined to perform step-by-step decoding on the smallest scale feature output by the feature extraction section. By using skip connections, the input features are combined with the output features of the feature extraction part to obtain a fused image.

2. The polarization image fusion system according to claim 1, characterized in that, Both the intensity coding branch and the polarization coding branch include: The shallow feature extraction module performs shallow feature extraction on the input image; The Swin Transformer module performs a moving window attention operation on shallow features to obtain the corresponding intensity features. The gradient extraction module performs dense residual operations on shallow features and calculates the gradient information of shallow features using gradient operators. The gradient information is then combined with the output features of the residual operations to obtain the corresponding gradient features.

3. The polarization image fusion system according to claim 1, characterized in that, The fusion unit consists of multiple cascaded affine coupling layers; in each affine coupling layer, a mapping function is used to perform bidirectional mapping on the input encoded features and gradient features; the encoded features and gradient features output by the last affine coupling layer are integrated to obtain the coupled features.

4. The polarization image fusion system according to claim 3, characterized in that, Each affine coupling layer includes a first mapping function and a second mapping function. Both the first and second mapping functions perform multiple consecutive convolution operations on the input features. Along the data processing direction, the number of channels in the first mapping function's convolution gradually increases, while the number of channels in the second mapping function's convolution gradually decreases. The forward mapping in each affine coupling layer is as follows: ； ； in, This represents the gradient feature output by the i-th affine coupling layer. Let represent the encoded features output by the i-th affine coupling layer, MF1 represent the first mapping function, MF2 represent the second mapping function, and exp represent the exponential function; The inverse mapping in each affine coupling layer is as follows: ； 。 5. The polarization image fusion system according to claim 1, characterized in that, The process of multidimensional convolution attention operations includes: The query matrix, key matrix, and value matrix are extracted from the input features. After performing matrix reshaping operations on the query matrix and key matrix respectively, the two features are combined and then subjected to a Softmax operation to obtain the transposed attention map. The transposed attention map is combined with the value matrix, and then summed pixel by pixel with the input features to obtain the output features.

6. The polarization image fusion system according to claim 5, characterized in that, The dynamic feedforward operation process includes: The input features are subjected to parallel convolution operations. The GELU activation operation is used to perform non-linear activation on the feature after one of the convolution operations to obtain the gate weights. These weights are then multiplied by the feature after the other convolution operation and added pixel by pixel to obtain the output features.

7. A polarization image fusion method, characterized in that, include: S1: Obtain a dataset containing linear polarization images, corresponding intensity images, and real fusion images of the two, and use it to create semantic segmentation labels for the segmentation task to obtain the training set; S2: Construct the polarization image fusion system as described in any one of claims 1 to 6, and train the polarization image fusion system using the training set obtained in step S1 to obtain a polarization image fusion model; during the training process, use the content loss and semantic loss between the predicted fused image output by the polarization image fusion model and the corresponding real fused image to train the polarization image fusion model. S3: Input the linear polarization degree image to be fused and its corresponding intensity image into the polarization image fusion model trained in step S2 to obtain the final fused image.

8. The polarization image fusion method according to claim 7, characterized in that, In step S2, the polarization image fusion model is trained using the composite loss function as follows: L=L content +β(m-1)×L semantic ; Where L represents the composite loss function, L content Indicates content loss, L semantic Let represent semantic loss, β represent the balance parameter between semantic loss and content loss, and m represent the number of training iterations; The content loss is obtained by the following formula: ； Where H and W represent the height and width of the fused image, respectively, and I F This represents the predicted fused image, S0 represents the intensity image, and DoLP represents the linear polarization degree image. This represents the gradient operator, where α represents the content weight; The process of calculating semantic loss includes: The real fused image and the predicted fused image are input into the pre-trained real-time semantic segmentation network to obtain their respective semantic segmentation results. The semantic loss is calculated using the following formula: ； Where h, w, and c represent the indices of the image's height, width, and channel directions, respectively, and L S0 The segmentation label L represents the image fused from the real data. s The transformed binary vector, I s I represents the semantic segmentation result corresponding to the predicted fused image. sa λ represents the auxiliary segmentation result corresponding to the predicted fused image, and λ represents the semantic loss weight.

Images