Polarization image fusion method based on channel cross neural network
By introducing a channel cross-neural network into polarized image fusion, a fusion network containing multiple cross-and attention modules is constructed, which solves the problem of poor results of existing methods in complex scenarios, and achieves more effective multi-dimensional feature fusion and significance Sexual information extraction.
Patent Information
- Application Number
- CN202411558535.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-21
AI Technical Summary
The existing polarized image fusion method has poor effect in complex fusion scenarios, and it is difficult to effectively fuse multi-dimensional features extracted from different source images, and lacks effective extraction and fusion of spatial information and channel information during feature extraction.
A polarized image fusion method based on channel cross neural network is proposed. By constructing a fusion network containing N fusion modules and one multi-layer convolution module, the channel cross module, packet convolution module and spatial attention module are used to perform feature crossing. and fusion, combining multi-scale structural similarity loss, strength loss and gradient loss for network training.
Effectively extract the significance information in image features, realize the cross-fusion of different source image features, and improve the fusion effect, especially the performance in complex environment contexts.
Smart Images
Figure CN119495006A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of computer vision, image processing and analysis, and in particular relates to a polarization image fusion method based on a channel cross neural network. Background Art
[0002] In traditional optical imaging technology, the polarization imaging system obtains the polarization information of the target by collecting the polarization image of the target, and obtains the light intensity distribution information in two-dimensional space by solving. The polarization image can describe the polarization characteristics of the target, such as the surface roughness, texture characteristics and shape contour of the target, but the overall light intensity is low; the intensity image can describe the reflectivity and transmittance of the target, and the overall light intensity is high. By fusing the polarization image and the intensity image using the polarization image fusion method, a fused image containing more dimensional information can be obtained, which is conducive to human visual observation and computer processing and analysis.
[0003] At present, traditional polarization fusion methods mainly include fusion algorithms based on multi-scale analysis and fusion algorithms based on representation learning. Fusion algorithms based on multi-scale analysis usually use different decomposition methods to decompose the source image into high-frequency and low-frequency parts or basic and detailed parts, and then design fusion rules for fusion. Fusion algorithms based on representation learning usually use the learned overcomplete dictionary to decompose the input data into a sparse coefficient matrix, and then select appropriate fusion rules for fusion. These methods usually require manual design of fusion rules. On the one hand, it is difficult to consider the feature differences between different input images, and on the other hand, they cannot effectively meet complex fusion requirements.
[0004] With the continuous development and application of deep learning, various neural network models and structures have been applied to image fusion to make up for the shortcomings of traditional fusion methods. The image fusion network model framework usually uses a pre-trained model as an encoder to extract input image features, designs feature fusion rules according to the specific requirements of the task, and finally uses a decoder to obtain a fused image. In addition, there are also fusion network model frameworks that integrate the fusion method into the network structure and implement model training by designing a loss function. Compared with traditional methods, the fusion network model based on deep learning can extract more accurate image features, and the end-to-end network model structure can improve the efficiency of image fusion. At present, image fusion network model frameworks with CNN, ResNet, GAN, Transformer, etc. as the backbone have achieved good results in polarization image fusion tasks, but there are still some shortcomings in the existing methods:
[0005] 1. The pre-trained neural network model can effectively extract the features of the input image, but some network fusion rules still need to be manually formulated, and the model fusion effect is poor in complex fusion scenarios.
[0006] 2. The fusion using network model structure is generally achieved through splicing and convolution operations, which makes it difficult to effectively fuse the multi-dimensional features extracted from different source images.
[0007] 3. Not every region of the source image needs to be fused. It is usually necessary to retain the spatial information of the salient target as much as possible. The extracted multi-dimensional features are not equally weighted for fusion. The channel information of the key features needs to be retained. The current fusion network lacks effective extraction and fusion of spatial and channel information in the feature extraction process, which reduces the fusion performance in complex environments. Summary of the invention
[0008] In order to solve the shortcomings of the above technology, the present invention proposes a polarization image fusion method based on a channel cross neural network, and the following steps are performed:
[0009] Step 1: Obtain N* groups of target polarization images The polarization images of the i-th polarization angles are 0°, 45°, 90°, and 135° respectively. The intensity image is calculated using the Stokes formula of formula (1): and linear polarization image DoLP i ,i∈N * ; The i-th intensity image and the i-th linear polarization image are calculated by the following formula:
[0010]
[0011]
[0012] Step 2: construct a polarization image fusion network based on channel cross-talk, which includes N fusion modules and 1 multi-layer convolution module;
[0013] Each fusion module contains two feature extraction branches with the same structure and one cross-connection branch;
[0014] The feature extraction branch has the source image as input in the first fusion module, and the image feature tensors output by the previous fusion module are input in the second to Nth fusion modules. The feature extraction branch includes 1 convolution layer, 1 ReLU activation function, 1 convolution layer, 1 ReLU activation function, and 1 SE module in sequence.
[0015] The SE module contains 1 convolution layer, 1 ReLU activation function, 1 average pooling layer, 1 convolution layer, 1 ReLU activation function, 1 convolution layer, 1 Sigmoid activation function, and 1 LayerNorm layer in sequence; there is a residual link from the first ReLU activation function to the LayerNorm layer;
[0016] The cross-connection branch contains 1 channel cross module CCM, 1 group convolution module GCM, and 1 spatial attention module SAM;
[0017] The input of the CCM module is the feature tensor obtained before the SE module in the two feature extraction branches. Represents a real feature tensor with data size C×H×W, and obtains the output Represents a real feature tensor of data size 2C×H×W, where T ′ The channel order is T 1 ,T 2 The channels cross each other, expressed as:
[0018]
[0019] In the formula, k and i are both positive natural numbers, T ′ (i,H,W) represents the feature tensor T ′ The i-th two-dimensional tensor in the first dimension, T 1 (k,H,W) represents the feature tensor T 1 The kth two-dimensional tensor in the first dimension, T 2 (k,H,W) represents the feature tensor T 2 The k-th 2D tensor in the first dimension of ;
[0020] The input of the GCM module is the output of the CCM module, which contains 1 group convolution layer and 1 ReLU activation function in sequence;
[0021] The input of SAM is the output of the GCM module. In SAM, the input passes through one channel average layer and one channel maximum layer respectively, and then is concatenated according to the channel dimension. Then it passes through one convolution layer, and finally passes through one Sigmoid layer to obtain the spatial attention feature tensor, which is dot-multiplied with the input of SAM and output as the total output of SAM.
[0022] The feature tensors output by SAM are added to the output feature tensors of the SE modules in the two feature extraction branches, and used as the inputs of the two feature extraction branches in the next fusion module.
[0023] The multi-layer convolution module consists of 3 convolutional layers and 2 ReLU activation functions. There is a ReLU activation function after the first and second convolutional layers, and the third convolutional layer directly outputs;
[0024] Step 3: construct a loss function Loss for training the polarization image fusion network; the loss function is expressed as:
[0025] Loss=Loss msssim +λ 1 Loss intensity +λ 2 Loss grad (4)
[0026] In formula (4), Loss msssim is the multi-scale structural similarity loss, Loss intensity is the strength loss, Loss grad is the gradient loss, λ 1 and λ 2 is the weight parameter;
[0027] Multi-scale structural similarity loss msssim It is expressed as:
[0028]
[0029] In formula (5), Loss ssim (I DoLP , I f ;ω s )and are structural similarity losses, is the weight coefficient, ω represents the local area, ω s Represents a local area of the image with a size of s×s; I DoLP is the input linear polarization image, is the input intensity image, I f is the output image after fusion;
[0030] Structural similarity loss ssim It is expressed as:
[0031]
[0032] In formula (6), and are local regions in image x respectively. The mean and variance of express Multiply and are the local regions in image y The mean and variance of for and The covariance coefficient, C 1 and C 2 are constants respectively;
[0033] Weight coefficient It is expressed as:
[0034]
[0035] In formula (7), g(x) = max(x, 0.0001), Represents image S respectively 0 and the variance of the local area in the image DOLP;
[0036] Strength loss intensity It is expressed as:
[0037]
[0038] In formula (8), μ is the proportional coefficient, H and W represent the length and width of the input feature tensor respectively, ‖‖ 1 represents the L1 norm;
[0039] Gradient Loss grad It is expressed as:
[0040]
[0041] In formula (9), represents the Sobel operator;
[0042] The Sobel operator uses two 3×3 convolution kernels to perform convolution calculation on the input image T to obtain the horizontal and vertical image grayscale values G x and G y , expressed as:
[0043]
[0044]
[0045] The gray value of the entire pixel of the image is expressed as:
[0046] G=|G x |+|G y | (12) Where |.| is the absolute value symbol;
[0047] Step 4: Based on the N sets of intensity images obtained in step 1 and linear polarization image DoLP i ,i∈N * , the gradient descent method is used to train the network model, and the loss function Loss is calculated to update the network parameters. When the set number of iterations is reached, the training is stopped, and the polarization image fusion network model is obtained to achieve effective fusion of polarization images.
[0048] In step 1 of a specific embodiment of the present invention, the input image size is 128×128.
[0049] In step 2 of another specific embodiment of the present invention, N=3.
[0050] In step 2 of another specific embodiment of the present invention, in the first convolutional layer of the feature extraction branch in sequence, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0; in the second convolutional layer, the convolution kernel size is 3×3, the number is 128, the step size is 1, and the padding is 1.
[0051] In step 2 of another specific embodiment of the present invention, in the SE module, in the first convolution layer in sequence, the convolution kernel size is 3×3, the number is 128, the step size is 1, and the padding is 1; in the first average pooling layer, the pooling output size is 1×1; in the second convolution layer, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0; in the third convolution layer, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0.
[0052] In addition, in step 2 of a specific embodiment of the present invention, in the first grouped convolution layer included in the GCM module input in sequence, the convolution kernel size is 3×3, the number is 128, the step size is 1, the padding is 1, and the number of groups is 128 channels.
[0053] In addition, in step 2 of a specific embodiment of the present invention, in one convolutional layer of SAM, the convolution kernel size is 7×7, the number is 128, the step size is 1, and the padding is 3.
[0054] In step 2 of another specific embodiment of the present invention, in the multi-layer convolution module, the convolution kernel size of the 1st to 3rd convolution layers is 3×3, the step size is 1, the padding is 1, and the number of convolution kernels is 64, 32, and 1 respectively.
[0055] In step 3 of another specific embodiment of the present invention, λ 1 =1,λ 2 =1; C 1 =0.0001, C 2 =0.0009.
[0056] Finally, in a specific embodiment of the present invention, the Adam optimizer is used for training; the number of iterations is 500; the learning rate is dynamically adjusted, with an initial value of 0.001, and the learning rate is multiplied by 0.6 at the 20th, 40th, 80th, 120th, 180th, 250th, 350th, and 450th rounds, respectively; the data set is a combination of public polarization data sets, which includes a total of 114 polarization image groups including 0°, 45°, 90°, and 135° directions: combination 1 includes 74 groups, and combination 2 includes 40 groups, of which 91 groups are used for training and 23 groups are used for testing: combination 1 includes 15 groups, and combination 2 includes 8 groups; using the image cropping method, 9 non-repeating images with a size of 128×128 are cropped from each image in the training set to construct a training set containing N=819 groups of images; in the test set, the size of combination 1 is 1024×1224, and the size of combination 2 is 1024×768.
[0057] The present invention can effectively extract significant information from image features, and simultaneously realize cross-fusion of features of images from different sources, further improving the fusion effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0058] Figure 1 Schematic diagram of the polarization image fusion method based on channel cross neural network in the present invention;
[0059] Figure 2 It is a schematic diagram of the structure of the fusion module in the polarization image fusion network based on channel crossover in the present invention;
[0060] Figure 3 It is a schematic diagram of the cross-connected branch structure in the fusion module of the present invention;
[0061] Figure 4 This is a comparison result diagram of combination 1 of the method in the present invention and other image fusion methods in the test set;
[0062] Figure 5 This is a comparison result diagram of combination 2 of the method in the present invention and other image fusion methods in the test set. DETAILED DESCRIPTION
[0063] The present invention will be described in detail below with reference to the accompanying drawings.
[0064] In one embodiment of the present invention, a polarization image fusion method based on a channel cross neural network is as follows: Figure 1 As shown, follow these steps:
[0065] Step 1: Get N * Group target polarization image The polarization images of the i-th polarization angles are 0°, 45°, 90°, and 135° respectively. The intensity image is calculated using the Stokes formula of formula (1): and linear polarization image DoLP i ,i∈N * Specifically, the i-th intensity image and the i-th linear polarization image are calculated by the following formulas:
[0066]
[0067] In a specific embodiment of the present invention, the input image size is 128×128.
[0068] Step 2: construct a polarization image fusion network based on channel cross-talk, including N fusion modules and 1 multi-layer convolution module; in this embodiment, N=3.
[0069] Each fusion module contains two feature extraction branches with the same structure and one cross-connected branch.
[0070] Except for the first fusion module, the input of the feature extraction branch is the source image. The inputs of the second to Nth fusion modules are all the image feature tensors output by the previous fusion module. The feature extraction branch includes 1 convolution layer (in this embodiment, the convolution kernel size is 1×1, the number is 128, the step length is 1, and the padding is 0), 1 ReLU activation function, 1 convolution layer (in this embodiment, the convolution kernel size is 3×3, the number is 128, the step length is 1, and the padding is 1), 1 ReLU activation function, and 1 SE module in sequence. Figure 2 The structure of the feature extraction branch is well known to those skilled in the art and will not be described again.
[0071] The SE module includes, in sequence, 1 convolution layer (in this embodiment, the convolution kernel size is 3×3, the number is 128, the step size is 1, and the padding is 1), 1 ReLU activation function, 1 average pooling layer (in this embodiment, the pooling output size is 1×1), 1 convolution layer (in this embodiment, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0), 1 ReLU activation function, 1 convolution layer (in this embodiment, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0), 1 Sigmoid activation function, and 1 LayerNorm layer, as shown in FIG. Figure 2 As shown in FIG. 1 , there is a residual link from the first ReLU activation function to the LayerNorm layer. The structure of the SE module is well known to those skilled in the art and will not be described again.
[0072] The cross-connection branch contains 1 channel cross module CCM, 1 group convolution module GCM, and 1 spatial attention module SAM.
[0073] The input of the CCM module is the feature tensor obtained before the SE module in the two feature extraction branches. Represents a real feature tensor with data size C×H×W, and obtains the output Represents a real feature tensor of data size 2C×H×W, where T ′ The channel order is T 1 ,T 2 The channels cross each other, which can be expressed as:
[0074]
[0075] In the formula, k and i are both positive natural numbers, T ′ (i,H,W) represents the feature tensor T ′ The i-th two-dimensional tensor in the first dimension, T 1 (k,H,W) represents the feature tensor T 1 The kth two-dimensional tensor in the first dimension, T 2 (k,H,W) represents the feature tensor T 2 The k-th 2-D tensor in the 1st dimension of .
[0076] The input of the GCM module is the output of the CCM module, which sequentially includes 1 grouped convolution layer (in one embodiment of the present invention, the convolution kernel size is 3×3, the number is 128, the step size is 1, the padding is 1, and the number of groups is the number of channels 128), and 1 ReLU activation function.
[0077] The input of SAM is the output of the GCM module. In SAM, the input passes through one channel average layer and one channel maximum layer respectively and then is spliced according to the channel dimension (splicing according to the channel dimension is well known to those skilled in the art), and then passes through one convolution layer (in this embodiment, the convolution kernel size is 7×7, the number is 128, the step size is 1, and the padding is 3), and finally passes through one Sigmoid layer to obtain the spatial attention feature tensor, which is point-multiplied with the input of SAM and output as the total output of SAM.
[0078] The feature tensors output by SAM are added to the output feature tensors of the SE modules in the two feature extraction branches and used as the inputs of the two feature extraction branches in the next fusion module.
[0079] The multi-layer convolution module consists of 3 convolution layers and 2 ReLU activation functions. There is a ReLU activation function after the 1st and 2nd convolution layers, and the 3rd convolution layer directly outputs. In this embodiment, the convolution kernel size of the 1st to 3rd convolution layers is 3×3, the step size is 1, the padding is 1, and the number of convolution kernels is 64, 32, and 1 respectively.
[0080] Step 3: Construct the loss function Loss for training the polarization image fusion network. The loss function is expressed as:
[0081] Loss=Loss msssim +λ 1 Loss intensity +λ 2 Loss grad (4)
[0082] In formula (4), Loss msssim is the multi-scale structural similarity loss, Loss intensity is the strength loss, Loss grad is the gradient loss, λ 1 and λ 2 is a weight parameter. 1 =1,λ 2 =1.
[0083] Multi-scale structural similarity loss msssim It is expressed as:
[0084]
[0085] In formula (5), Loss ssim (I DoLP , I f ;ω s )and I DoLP with I f and with I f The structural similarity loss is is the weight coefficient, ω is the abbreviation of windows, representing the local area, ω s Represents a local area of the image with a size of s×s; I DoLP is the input linear polarization image, is the input intensity image, I f is the output image after fusion.
[0086] Structural similarity loss ssim It is expressed as:
[0087]
[0088] In formula (6), and are local regions in image x respectively. The mean and variance of express Multiply and are the local regions in image y The mean and variance of for and The covariance coefficient, C 1 and C 2 are constants respectively. In this embodiment, C 1 =0.0001, C 2 =0.0009.
[0089] Weight coefficient It is expressed as:
[0090]
[0091] In formula (7), g(x) = max(x, 0.0001), Represents image S respectively 0 and the variance of the local region in the image DOLP.
[0092] Strength loss intensity It is expressed as:
[0093]
[0094] In formula (8), μ is the proportional coefficient, H and W represent the length and width of the input feature tensor respectively, ‖‖ 1 represents the L1 norm.
[0095] Gradient Loss grad It is expressed as:
[0096]
[0097] In formula (9), represents the Sobel operator.
[0098] The Sobel operator uses two 3×3 convolution kernels to perform convolution calculation on the input image T to obtain the horizontal and vertical image grayscale values G x and G y , expressed as:
[0099]
[0100]
[0101] The gray value of the entire pixel of the image is expressed as:
[0102] G=|G x |+|G y | (12)Wherein, |.| is the absolute value symbol.
[0103] Step 4: Based on the N sets of intensity images obtained in step 1 and linear polarization image DoLP i ,i∈N * , the gradient descent method is used to train the network model, and the loss function Loss is calculated to update the network parameters. When the set number of iterations is reached, the training is stopped, and the polarization image fusion network model is obtained to achieve effective fusion of polarization images.
[0104] In one embodiment of the present invention, the training adopts the Adam optimizer; the number of iterations is 500; the learning rate adopts a dynamic adjustment method, the initial value is 0.001, and the learning rate is multiplied by 0.6 at the 20th, 40th, 80th, 120th, 180th, 250th, 350th, and 450th rounds respectively; the data set is a combination of public polarization data sets, which contains a total of 114 polarization image groups including 0°, 45°, 90°, and 135° directions (combination 1 contains 74 groups, and combination 2 contains 40 groups), of which 91 groups are used for training and 23 groups are used for testing (combination 1 contains 15 groups, and combination 2 contains 8 groups); using the image cropping method, each image in the training set is cropped into 9 non-repeating images of size 128×128, and a training set containing 819 groups (N=819) of images is constructed. In the test set, the size of combination 1 is 1024×1224, and the size of combination 2 is 1024×768.
[0105] Table 1 Comparison of experimental results of different fusion methods (combination 1)
[0106]
[0107]
[0108] Table 2 Comparison of experimental results of different fusion methods (combination 2)
[0109]
[0110] The evaluation indicators in Tables 1 and 2 are information entropy EN, spatial frequency SF, standard deviation SD, peak signal-to-noise ratio PSNR, mean square error MSE, mutual information MI, visual fidelity VIF, difference correlation and SCD, gradient-based fusion performance Qabf, multi-scale structural similarity measurement MS_SSIM, and pixel feature mutual information FMI_pixel.
[0111] Tables 1 and 2 show the test results of the method of the present invention compared with the traditional fusion methods (CVT, GTF), CNN-based fusion methods (DenseNet, U2Fusion, IFCNN) and Transformer-based fusion methods (SwinFusion) under the test set data of combination 1 and combination 2, respectively, where the underline represents the top 2 best values, and the bold represents the best value.
Claims
1. A polarization image fusion method based on channel cross neural network, characterized in that: The method proceeds as follows: Step 1: Get N * Group target polarization image The polarization images of the i-th polarization angles are 0°, 45°, 90°, and 135° respectively. The intensity image is calculated using the Stokes formula of formula (1): i∈N * and linear polarization image DoLP i ,i∈N * ; The i-th intensity image and the i-th linear polarization image are calculated by the following formula: Step 2: construct a polarization image fusion network based on channel cross-talk, which includes N fusion modules and 1 multi-layer convolution module; Each fusion module contains two feature extraction branches with the same structure and one cross-connection branch; The feature extraction branch has the source image as input in the first fusion module, and the image feature tensors output by the previous fusion module are input in the second to Nth fusion modules. The feature extraction branch includes 1 convolution layer, 1 ReLU activation function, 1 convolution layer, 1 ReLU activation function, and 1 SE module in sequence. The SE module contains 1 convolution layer, 1 ReLU activation function, 1 average pooling layer, 1 convolution layer, 1 ReLU activation function, 1 convolution layer, 1 Sigmoid activation function, and 1 LayerNorm layer in sequence; there is a residual link from the first ReLU activation function to the LayerNorm layer; The cross-connection branch contains 1 channel cross module CCM, 1 group convolution module GCM, and 1 spatial attention module SAM; The input of the CCM module is the feature tensor T1 obtained before the SE module in the two feature extraction branches. Represents a real feature tensor with data size C×H×Q, and obtains the output Represents a real feature tensor of data size 2C×H×W, where T ′ The channel order is that T1 and T2 channels are crossed in pairs, expressed as: In the formula, k and i are both positive natural numbers, T ′ (i,H,W) represents the feature tensor T ′ T1(k,H,W) represents the i-th two-dimensional tensor in the first dimension of the feature tensor T1, and T2(k,H,W) represents the k-th two-dimensional tensor in the first dimension of the feature tensor T2; The input of the GCM module is the output of the CCM module, which contains 1 group convolution layer and 1 ReLU activation function in sequence; The input of SAM is the output of the GCM module. In SAM, the input passes through one channel average layer and one channel maximum layer respectively, and then is concatenated according to the channel dimension. Then it passes through one convolution layer, and finally passes through one Sigmoid layer to obtain the spatial attention feature tensor, which is dot-multiplied with the input of SAM and output as the total output of SAM. The feature tensors output by SAM are added to the output feature tensors of the SE modules in the two feature extraction branches, and used as the inputs of the two feature extraction branches in the next fusion module. The multi-layer convolution module consists of 3 convolutional layers and 2 ReLU activation functions. There is a ReLU activation function after the first and second convolutional layers, and the third convolutional layer directly outputs; Step 3, construct a loss function Loss for training the polarization image fusion network; The loss function is expressed as: Loss=Loss msssim +λ1Loss intensity +λ2Loss grad (4) In formula (4), Loss msssim is the multi-scale structural similarity loss, Loss intensity is the strength loss, Loss grad is the gradient loss, λ1 and λ2 are weight parameters; Multi-scale structural similarity loss msssim It is expressed as: In formula (5), Loss ssim (I DoLP ,I f ;ω s )and are structural similarity losses, is the weight coefficient, ω represents the local area, ω s I represents a local area of the image with a size of s×s; DoLP is the input linear polarization image, is the input intensity image, I f is the output image after fusion; Structural similarity loss ssim It is expressed as: In formula (6), and are local regions in image x respectively. The mean and variance of express Multiply and are the local regions in image y The mean and variance of for and The covariance coefficients of , C1 and C2 are constants respectively; Weight coefficient It is expressed as: In formula (7), g(x) = max(x, 0.0001), Represent the variance of the local area in image S0 and image DOLP respectively; Strength loss intensity It is expressed as: In formula (8), μ is the scaling coefficient, H and W represent the length and width of the input feature tensor, respectively, and ‖‖1 represents the L1 norm; Gradient Loss grad It is expressed as: In formula (9), represents the Sobel operator; The Sobel operator uses two 3×3 convolution kernels to perform convolution calculation on the input image T to obtain the horizontal and vertical image grayscale values G x and G y , expressed as: The gray value of the entire pixel of the image is expressed as: G=|G x |+|G y | (12) Among them, |.| is the absolute value symbol; Step 4: Based on the N sets of intensity images obtained in step 1 i∈N * and linear polarization image DoLP i ,i∈N * , the gradient descent method is used to train the network model, and the loss function Loss is calculated to update the network parameters. When the set number of iterations is reached, the training is stopped, and the polarization image fusion network model is obtained to achieve effective fusion of polarization images.
2. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 1, the input image size is 128×128.
3. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, N=3.
4. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, in the first convolutional layer of the feature extraction branch, the convolution kernel size is 1×1, the number is 128, the step size is 1, and the padding is 0; in the second convolutional layer, the convolution kernel size is 3×3, the number is 128, the step size is 1, and the padding is 1.
5. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, the SE module has the following sequence: in the first convolution layer, the convolution kernel size is 3×3, the number is 128, the stride is 1, and the padding is 1; in the 1st average pooling layer, the pooling output size is 1×1; in the second convolution layer, the convolution kernel size is 1×1, the number is 128, the stride is 1, and the padding is 0; in the third convolution layer, the convolution kernel size is 1×1, the number is 128, the stride is 1, and the padding is 0.
6. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, the GCM module input is included in the first grouped convolution layer in order, the convolution kernel size is 3×3, the number is 128, the step size is 1, the padding is 1, and the number of groups is 128 channels.
7. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, in one convolutional layer of SAM, the convolution kernel size is 7×7, the number is 128, the stride is 1, and the padding is 3.
8. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 2, in the multi-layer convolution module, the convolution kernel size of the 1st to 3rd convolution layers is 3×3, the step size is 1, the padding is 1, and the number of convolution kernels is 64, 32, and 1 respectively.
9. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: In step 3, λ1=1, λ2=1; C1=0.0001, C2=0.0009.
10. The polarization image fusion method based on channel cross neural network according to claim 1, characterized in that: The Adam optimizer was used for training; the number of iterations was 500; the learning rate was dynamically adjusted with an initial value of 0.001, and the learning rate was multiplied by 0.6 at the 20th, 40th, 80th, 120th, 180th, 250th, 350th, and 450th rounds, respectively; the dataset was a combination of public polarization datasets, containing a total of 114 polarization image groups with directions of 0°, 45°, 90°, and 135°: combination 1 contained 74 groups, and combination 2 contained 40 groups, of which 91 groups were used for training and 23 groups were used for testing: combination 1 contained 15 groups, and combination 2 contained 8 groups; using the image cropping method, 9 non-repeating images of size 128×128 were cropped from each image in the training set to construct a training set containing N=819 groups of images; in the test set, the size of combination 1 was 1024×1224, and the size of combination 2 was 1024×768.