An infrared polarization image fusion enhancement method based on a learnable low-rank representation

Abstract

The application discloses an infrared polarization image fusion enhancement method based on a learnable low-rank representation, and steps of the method comprise the following steps: 1, collecting infrared polarization images in different directions and calculating Stokes vectors; 2, constructing an infrared polarization image fusion network based on a learnable low-rank representation; 3, performing encoding and decoding reconstruction on the fused infrared polarization image and obtaining a polarization image after guided filtering; 4, establishing a loss function between the input infrared polarization image and the fused image; 5, training the network by using the infrared polarization image, and performing fusion enhancement on the infrared polarization image to be processed by using the trained model. The application realizes low-rank feature representation of the infrared polarization image by constructing an end-to-end deep learning network, can effectively extract significant target information in the polarization image, and reduces network model parameters, and meanwhile, a fusion loss function based on polarization feature reconstruction is designed, so that the background noise interference of the infrared polarization image can be inhibited.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, image preprocessing and analysis, and specifically relates to an infrared polarized image fusion enhancement method based on a learnable low-rank representation. BACKGROUND

[0002] Infrared polarized image fusion is a technical method of fusing infrared images and polarized images, which combines infrared thermal imaging and polarized light imaging technology to obtain more comprehensive and accurate target detection, recognition and tracking capabilities. The infrared polarized image fusion technology can effectively make up for the limitations of infrared and polarized images by fusing the two different information sources, and improve the accuracy and reliability of target detection and recognition. Specifically, infrared imaging can provide thermal distribution information of the target, which can be used to detect hidden targets and temperature changes of the target, etc. Polarized images can provide surface morphology and material characteristics of the target, which can be used to identify the edge and contour features of specific targets, especially man-made objects. Therefore, the infrared polarized image fusion technology can improve the detection and recognition accuracy of targets in complex backgrounds, and has broad application prospects in space exploration, military reconnaissance, disaster search, environmental monitoring and other fields.

[0003] Traditional image fusion algorithms are represented by multi-scale transformation, low-rank representation and sparse representation. These methods process different source images in the transformation domain, or perform feature decomposition based on low-rank and sparse representation mathematical models to obtain the saliency and detail features in the image. In specific scenarios, these methods have obvious advantages and good interpretability. However, due to the inherent iterative optimization process, this method is very time-consuming, which reduces the feasibility of practical application in many scenarios, and the cumbersome and inefficient fusion strategy cannot adapt to complex fusion scenarios.

[0004] Current deep learning methods based on CNN and Transformer show good fusion performance in the field of image fusion, and have achieved good results in some specific fusion tasks. However, the existing deep learning-based image fusion network has a large number of parameters, which limits the expansion of subsequent high-level computer vision tasks such as target detection. In addition, complex acquisition environments will cause a large amount of noise in the infrared polarization degree image. The existing fusion method generally does not consider the noise interference in the fusion process, and directly fuses the infrared intensity and polarization degree image, or introduces other polarization parameters as input to improve the fusion performance, but the complexity of the network also increases. SUMMARY

[0005] The present application is to solve the above-mentioned prior art existing deficiencies, put forward a kind of infrared polarized image fusion enhancement method based on learnable low rank representation, to be able to construct learnable low rank representation model to extract polarized significant target feature, and make the model lightweight, satisfy the real-time demand of network, simultaneously, in combination with polarized imaging principle further suppresses the background noise interference of infrared polarized image.

[0006] The present application is to solve the above-mentioned prior art existing deficiencies, put forward a kind of infrared polarized image fusion enhancement method based on learnable low rank representation, to be able to construct learnable low rank representation model to extract polarized significant target feature, and make the model lightweight, satisfy the real-time demand of network, simultaneously, in combination with polarized imaging principle further suppresses the background noise interference of infrared polarized image.

[0007] The present application is to solve the above-mentioned prior art existing deficiencies, put forward a kind of infrared polarized image fusion enhancement method based on learnable low rank representation, to be able to construct learnable low rank representation model to extract polarized significant target feature, and make the model lightweight, satisfy the real-time demand of network, simultaneously, in combination with polarized imaging principle further suppresses the background noise interference of infrared polarized image.

[0008] Step 1, obtain infrared polarized image, calculate polarized Stokes vector;

[0009] N groups of infrared polarized images containing 0 °, 45 °, 90 °, 135 ° direction are obtained respectively, and N groups of corresponding polarized Stokes vectors are calculated{{I1,Q1,U1}...,{I n ,Q n ,U n},...,{I N ,Q N ,U N}}. n ,...,I N}、{D1,...,D n ,...,D N}、{A1,...,A n ,...,A N} and {C1,...,C n ,...,C N} are obtained, wherein I n represents the first component of the n infrared intensity image, i.e. the n group of polarized Stokes vector, Q n And U n Second and third components of the n group of polarized Stokes vector, D represents the n polarized degree image, A represents the n polarized angle image,

[0010] Step 2, constructing an infrared polarization image fusion network based on a learnable low-rank representation, comprising: an infrared intensity low-rank feature encoding module Encoder_I, an infrared polarization low-rank feature encoding module Encoder_D, an attention-guided filter fusion module AGFF, and a fusion feature decoding module Decoder;

[0011] Step 2.1, the nth infrared intensity image I n After being input into Encoder_I for processing, the nth infrared intensity encoding feature tensor EN_I n is obtained.

[0012] Step 2.2, constructing Encoder_D with the same structure as Encoder_I, and inputting the nth polarization degree image D n into Encoder_D for processing, to obtain the nth polarization encoding feature tensor EN_D n .

[0013] Step 2.3, extracting the first half of EN_I n , i.e., the infrared intensity low-rank encoding feature tensor, denoted as L_I n , and the second half of EN_D n , i.e., the polarization sparse encoding feature tensor, denoted as S_D n .

[0014] Step 2.4, the attention-guided filter fusion module AGFF processes L_I n and S_D n to obtain the nth fused feature tensor IG n .

[0015] Step 2.5, the fusion feature decoding module Decoder is composed of H feature decoding units DeU and a regression output unit RU; each feature decoding unit DeU is composed of a convolution layer with a convolution kernel of 3x3 and a LeakyReLU activation layer, and the regression output unit RU is composed of a convolution layer with a convolution kernel of 3x3 and a Tanh activation layer.

[0016] The nth fused feature tensor IG n is input into the fusion feature decoding module Decoder for processing, and the nth fusion-enhanced infrared polarization image F n is output, thereby obtaining a set of fusion-enhanced infrared polarization images {F1,…, F n ,…, F N}.

[0017] Step 3, encode and reconstruct the fused infrared polarization image, and obtain the guided filtered polarization image;

[0018] Step 3.1, F n are input into Encoder_I and Encoder_D respectively, and the nth infrared intensity fusion encoding tensor EI n and the nth polarization fusion encoding tensor ED n are obtained respectively. n n EI n and ED n are input into Decoder respectively for decoding, and the nth reconstructed infrared intensity image RI n and the nth reconstructed infrared polarization degree image RD n are obtained respectively. n ;

[0019] Step 3.2, the nth infrared intensity image I n is taken as a guide image, and the nth polarization degree image D n and the nth polarization combination parameter image C n are guided filtered respectively, so as to obtain the nth guided filtered polarization degree image GD n and the nth guided filtered polarization combination parameter image GC n ;

[0020] Step 4, establish the loss function between the input infrared polarization image and the fusion image;

[0021] Step 4.1, construct the multi-scale structural similarity loss L MSSS using formula (4)

[0022]

[0023] In formula (4), MWSSIM(·) represents the multi-scale structural similarity loss function;

[0024] Step 4.2, construct the encoding reconstruction loss L REC using formula (5)

[0025]

[0026] In formula (5), MAE represents the mean absolute error loss function;

[0027] Step 4.3, calculate the fusion loss based on polarization feature reconstruction L POL using formula (6)

[0028]

[0029] In formula (6), respectively represent the reconstructed 0°, 60° and 120° direction polarized images of the nth image;

[0030] Step 4.4, constructing an infrared polarization image fusion enhanced total loss function L using formula (7) n :

[0031]

[0032] In formula (7), λ1, λ2, λ3 are three weight coefficients;

[0033] Step 5, based on the infrared intensity image set {I1,...,I n ,...,I N}, the polarization degree image set {D1,...,D n ,...,D N}, the polarization angle image set {A1,...,A n ,...,A N} and the polarization combination parameter image set {C1,...,C n ,...,C N}, the infrared polarization image fusion network is trained by using a gradient descent algorithm, and the total loss function L n is calculated to update the network parameters, and when the number of training iterations reaches a set number of times, the training is stopped, so that the trained infrared polarization image fusion enhancement model is obtained, which is used to realize the fusion and enhancement of infrared intensity and polarization degree images.

[0034] The infrared polarization image fusion enhancement method based on the learnable low-rank representation provided by the application is characterized in that the infrared intensity low-rank feature coding module Encoder_I in step 2.1 is composed of a convolution layer with a convolution kernel of 3*3, a LeakyReLU activation layer and M low-rank feature extraction modules in sequence, wherein the mth low-rank feature extraction module is denoted as LR-conv m , m∈[1,M];the mth low-rank feature extraction module LR-conv m includes a convolution layer with a convolution kernel of 1*1, a BN layer, a PReLU activation layer, and two parallel branches in sequence, wherein the first branch is composed of a single convolution layer with a convolution kernel of 3*3, and the second branch is composed of L+1 low-rank feature representation blocks in sequence; wherein the lth low-rank feature representation block of the mth low-rank feature extraction module LR-conv m is denoted as l∈[0,L];

[0035] The nth infrared intensity image I nThe input is fed into Encoder_I and processed sequentially through a 3×3 convolutional layer and a LeakyReLU activation layer to obtain the nth infrared precoding feature tensor IPinput. n ;

[0036] When m=1, the feature tensor IPinput n LR-conv, the m-th low-rank feature extraction module m The m-th input feature tensor is denoted as And through LR-conv m After processing with a 1×1 convolutional layer, a BN layer, and a PReLU activation layer, the m-th infrared convolutionally encoded feature tensor is obtained.

[0037] After processing by the first branch, the m-th infrared fundamental coded feature tensor is obtained.

[0038] When l = 0 As The l-th input feature tensor is denoted as After processing by a convolutional layer with a 1×1 kernel, the l-th feature tensor is obtained. Reusing the activation function h shown in equation (1) θ (·)right After processing, the l-th activated feature tensor is output. Recorded as The l-th output feature tensor

[0039]

[0040] In equation (1), sign(·) is the sign function, and θ is the threshold value;

[0041] When l = 1, 2, ..., L, the (l-1)th low-rank feature block is represented. The (l-1)th output feature tensor As The l-th input feature tensor And input Thus, by using equation (2), we can obtain The l-th output feature tensor Finally, the Lth low-rank feature block of the second branch is represented. Obtain the Lth output feature tensor

[0042]

[0043] In equation (2), conv1 represents the first convolutional layer with a 1×1 kernel, conv2 represents the second convolutional layer with a 1×1 kernel, and λ R The residual coefficient;

[0044] Output from the first branch and After concatenation, the data passes through a BN layer, a PReLU activation layer, and a Channel Attention layer in sequence to obtain the m-th infrared encoded aggregated feature tensor. Then it is compared with the m-th feature tensor After performing residual connection, the m-th infrared low-rank sparse feature tensor is output.

[0045] When m = 2, 3, ..., M, Input the m-th low-rank feature extraction module LR-conv m The process is performed to obtain the output feature tensor. Thus, the Mth low-rank feature extraction module LR-conv M Output feature tensor And it is used as the nth infrared intensity coding feature tensor of Encoder_I, denoted as EN_I. n .

[0046] The attention-guided filtering fusion module AGFF consists of an attention-guided filter block AGF and a feature fusion block FFB, and obtains the nth fused feature tensor IG according to the following process. n ;

[0047] Step 2.4.1, L_I n With S_D n The input is fed into the attention-guided filter block (AGF). It first passes through a 1×1 convolutional layer to obtain two feature tensors, which are then superimposed. Next, it passes through a ReLU activation layer, a 1×1 convolutional layer, and a Sigmoid activation layer to obtain the nth attention feature tensor T. n Finally, L_I n As a guide map, using T n Obtain the filter coefficients for use in S_D n Perform guided filtering to obtain the nth filtered feature tensor G. n ;

[0048] Step 2.4.2, transfer L_I n With G nAfter the concatenation operation, the input is input into the feature fusion block FFB, and is sequentially processed by the convolution layer with a convolution kernel of 3*3 and the Sigmoid activation layer, so that the obtained feature tensor is segmented in the channel dimension, and the first half of the feature tensor is denoted as FI n , and the second half of the feature tensor is denoted as FG n ; so that the nth fused feature tensor IG n is obtained through formula (3):

[0049]

[0050] In formula (4), denote the vector parameters to be learned with the same channel number as L_I n , G n , respectively.

[0051] The guided filtering operation in the step 2.4.1 is to obtain two linear coefficients of the local window centering on the pixel coordinate k, so as to perform a local linear weighting operation on the guided image L_I n according to to obtain the filtered feature tensor G n ;

[0052]

[0053] In formula (8), N k is the number of pixels in the local window , lambda is a regularization coefficient, T n (i) denotes the value of the ith pixel point of the nth attention feature tensor T n , L_I n (i) denotes the value of the ith pixel point of the nth infrared intensity low-rank encoding feature tensor L_I n , S_D n (i) denotes the value of the ith pixel point of the nth polarization sparse encoding feature tensor S_D n , and X n (i) denotes the value of the ith pixel point of the nth normalized attention feature tensor X n .

[0054] The electronic device comprises a memory and a processor, and the characteristic of the electronic device is that the memory is used for storing a program supporting the processor to execute the infrared polarization image fusion enhancement method, and the processor is configured to execute the program stored in the memory.

[0055] The application discloses a computer readable storage medium, and a computer program is stored on the computer readable storage medium.

[0056] Compared with the prior art, the application has the beneficial effects that:

[0057] 1、The application constructs a deep learning network model based on low rank representation, which can effectively extract the detailed information in the infrared intensity image and the salient target features in the polarization degree image, and compared with the existing fusion network, the model is more lightweight, effectively reduces the operation time of the model, and meets the real-time demand in a real scene.

[0058] 2、The application directly embeds the guided filtering method into the deep learning network, can realize adaptive filtering of the polarization degree image, and designs a learnable feature fusion block, compared with the traditional feature concatenation and superposition operation, the application can realize effective fusion between features of different source images, and meanwhile reduces the background noise interference in the fused image.

[0059] 3、The existing deep learning-based fusion method only utilizes the fused image and the input infrared polarization image to perform loss calculation, while the application designs a fusion loss function based on polarization feature reconstruction, reconstructs the infrared polarization features by inputting the fused image into a coding and decoding network, combines Malus law to restore the original Stokes vector, and thus calculates the polarization feature reconstruction loss, so that the method can avoid direct introduction of the loss calculation of the polarization degree, meanwhile retains the details of the infrared intensity image and the polarization salient target features, further suppresses the noise of the polarization degree image, and improves the fusion performance of the infrared polarization image in a complex scene. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 A flowchart of the infrared polarization image fusion enhancement method in the application;

[0061] Figure 2 A network structure diagram of the infrared polarization image fusion enhancement in the application;

[0062] Figure 3 A structure diagram of the low rank feature extraction module in the application;

[0063] Figure 4 A structure diagram of the low rank feature representation block in the application;

[0064] Figure 5 A structure diagram of the attention guided filtering fusion module in the application;

[0065] Figure 6 A structure diagram of the attention guided filtering block in the application. DETAILED DESCRIPTION

[0066] In this embodiment, an infrared polarization image fusion enhancement method based on learnable low-rank representation is used. By constructing an infrared polarization image fusion network based on low-rank representation, low-rank features in the infrared intensity image and sparse representation in the polarization degree image can be obtained respectively. In the feature fusion part, an attention-guided filtering fusion method is designed to reduce the noise interference of the polarization degree image. Finally, a fusion loss based on polarization feature reconstruction is introduced for training. Specifically, as shown in Figure 1 , the following steps are performed:

[0067] Step 1, obtain the infrared polarization image and calculate the polarization Stokes vector.

[0068] N groups of infrared polarization images containing 0°, 45°, 90°, and 135° directions are obtained respectively, and N groups of corresponding polarization Stokes vectors are calculated. n n n N N N Thus, the infrared intensity image set {I1,...,I n ,...,I N}, the polarization degree image set {D1,...,D n ,...,D N}, the polarization angle image set {A1,...,A n ,...,A N}, and the polarization combined parameter image set {C1,...,C n ,...,C N} are obtained, where I n represents the nth infrared intensity image, i.e., the first component of the nth group of polarization Stokes vectors, Q n and U n represent the second and third components of the nth group of polarization Stokes vectors, respectively, represents the nth polarization degree image, represents the nth polarization angle image, represents the nth polarization combined parameter image.

[0069] Step 2, construct an infrared polarization image fusion network based on learnable low-rank representation. The structure of the infrared polarization image fusion enhancement network is as shown in Figure 2 ​​​​​As shown, it comprises: an infrared intensity low-rank feature encoding module Encoder_I, an infrared polarization low-rank feature encoding module Encoder_D, an attention guided filter fusion module AGFF, and a fusion feature decoding module Decoder.

[0070] Step 2.1, the infrared intensity low-rank feature encoding module Encoder_I is sequentially composed of a convolution layer with a convolution kernel of 3x3, a LeakyReLU activation layer, and M low-rank feature extraction modules, wherein the mth low-rank feature extraction module is denoted as LR-conv m , m∈[1,M]; in this embodiment, the number of low-rank feature extraction modules M=2; the structure of the low-rank feature extraction module LR-conv m is shown in Figure 3 , the mth low-rank feature extraction module LR-conv m comprises a convolution layer with a convolution kernel of 1x1, a BN layer, a PReLU activation layer, and two parallel branches in sequence, wherein the first branch is composed of a single convolution layer with a convolution kernel of 3x3, and the second branch is sequentially composed of L+1 low-rank feature representation blocks; wherein the lth low-rank feature representation block of the mth low-rank feature extraction module LR-conv m is denoted as l∈[0,L]; in this embodiment, the number of low-rank feature representation blocks is 4, L=3, and the structure of the low-rank feature representation block is shown in Figure 4 ;

[0071] The nth infrared intensity image I n is input into Encoder_I and sequentially processed by a 3x3 convolution layer and a LeakyReLU activation layer to obtain the nth infrared pre-encoding feature tensor IPinput n ;

[0072] When m=1, the feature tensor IPinput n is taken as the mth input feature tensor of the mth low-rank feature extraction module LR-conv m , denoted as , and processed by the convolution layer with a convolution kernel of 1x1, the BN layer, and the PReLU activation layer in LR-conv m , to obtain the mth infrared convolutional encoding feature tensor

[0073] After processing by the first branch, the mth infrared basic encoding feature tensor

[0074] When l=0, is taken as the lth input feature tensor of , denoted as After processing by a convolutional layer with a 1×1 kernel, the l-th feature tensor is obtained. Reusing the activation function h shown in equation (1) θ (·)right After processing, the l-th activated feature tensor is output. Recorded as The l-th output feature tensor

[0075]

[0076] In equation (1), sign(·) is the sign function, and θ is the threshold value;

[0077] When l = 1, 2, ..., L, the (l-1)th low-rank feature block is represented. The (l-1)th output feature tensor As The l-th input feature tensor And input Thus, by using equation (2), we can obtain The l-th output feature tensor Finally, the Lth low-rank feature block of the second branch is represented. Obtain the Lth output feature tensor

[0078]

[0079] In equation (2), conv1 represents the first convolutional layer with a 1×1 kernel, conv2 represents the second convolutional layer with a 1×1 kernel, and λ R The residual coefficient is λ. In this embodiment, the low-rank decomposition and sparse representation of the input infrared polarization features are achieved through equation (2), and the residual coefficient λ is λ. R It updates automatically through network learning;

[0080] Output from the first branch and After concatenation, the data passes through a BN layer, a PReLU activation layer, and a Channel Attention layer in sequence to obtain the m-th infrared encoded aggregated feature tensor. Then it is compared with the m-th feature tensor After performing residual connection, the m-th infrared low-rank sparse feature tensor is output.

[0081] When m = 2, 3, ..., M, Input the m-th low-rank feature extraction module LR-conv mThe process is performed to obtain the output feature tensor. Thus, the Mth low-rank feature extraction module LR-conv M Output feature tensor And it is used as the nth infrared intensity coding feature tensor of Encoder_I, denoted as EN_I. n ;

[0082] Step 2.2: Construct Encoder_D with the same structure as Encoder_I, and then convert the nth polarization image D... n After processing the input into Encoder_D, the nth polarization-coded feature tensor EN_D is obtained. n ;

[0083] Step 2.3: Extract EN_I from the channel dimension n The first half of the feature tensor, namely the infrared intensity low-rank encoded feature tensor, is denoted as L_I. n Simultaneously extract EN_D in the channel dimension n The latter half of the feature tensor, namely the polarization sparse coding feature tensor, is denoted as S_D. n In this embodiment, it is necessary to extract detailed information from the infrared intensity image and salient target features from the polarization degree image, respectively. Using Encoder_D and Encoder_I, the low-rank and sparse coefficients of the input image can be obtained simultaneously. Therefore, this invention extracts EN_I separately. n The low-rank coefficients in EN_D n The sparse coefficients in the image are: low-rank coefficients reflect basic detail features in the image, while sparse coefficients represent salient target information in the image.

[0084] Step 2.4: The attention-guided filter fusion module AGFF consists of the attention-guided filter block AGF and the feature fusion block FFB; the structure of the attention-guided filter fusion module is as follows: Figure 5 As shown;

[0085] Step 2.4.1, L_I n With S_D n The input is fed into the attention-guided filter block (AGF), whose structure is as follows: Figure 6 As shown, to reduce noise interference in polarization images, this invention embeds guided filtering into a deep learning network and adjusts the linear coefficients of the guided filter using the generated attention feature tensor, thereby achieving adaptive filtering of the polarization image by infrared intensity. First, the tensors are processed by a 1×1 convolutional layer to obtain two feature tensors, which are then superimposed. Next, they are processed sequentially by a ReLU activation layer, a 1×1 convolutional layer, and a Sigmoid activation layer to obtain the nth attention feature tensor T.n Finally, L_I n As a guide map, using T n Obtain the filter coefficients for S_D n Perform guided filtering to obtain the nth filtered feature tensor G. n ;

[0086] In practice, the guided filtering operation uses equation (3) to obtain a local window with pixel coordinates centered at k. Two linear coefficients Therefore, according to For the guide graph L_I n Perform a local linear weighting operation to obtain the filtered feature tensor G. n ;

[0087]

[0088] In equation (8), N k For local windows The number of pixels in T, λ is the regularization coefficient, and T n (i) represents the nth attention feature tensor T n The value of the i-th pixel, L_I n (i) represents the nth infrared intensity low-rank encoded feature tensor L_I n The value of the i-th pixel, S_D n (i) represents the nth polarization sparse coding feature tensor S_D n The value of the i-th pixel, X n (i) represents the nth normalized attention feature tensor X. n The value of the i-th pixel.

[0089] Step 2.4.2, transfer L_I n With G n After concatenation, the input is fed into the feature fusion block FFB, and then processed sequentially through a 3×3 convolutional layer and a sigmoid activation layer. The resulting feature tensor is split along the channel dimension, and the first half of the feature tensor is denoted as FI. n The second half of the feature tensor is denoted as FG. n Thus, the nth fused feature tensor IG is obtained through equation (4). n :

[0090] IG n =L_I n ×FI n ×V n I +G n ×FG n ×Vn G (4)

[0091] In formula (4), V n G respectively, and L_I n , G n have the same number of channels to be learned vector parameters;

[0092] Step 2.5, the fusion feature decoding module Decoder is composed of H feature decoding units DeU and a regression output unit RU; in this embodiment, the number of feature decoding units DeU H = 3; wherein each feature decoding unit DeU is composed of a convolution layer with a convolution kernel of 3x3, a LeakyReLU activation layer, and the regression output unit RU is composed of a convolution layer with a convolution kernel of 3x3, and a Tanh activation layer;

[0093] The n-th fused feature tensor IG n is input into the fusion feature decoding module Decoder for processing, and the n-th fusion enhanced infrared polarization image F n is output, so as to obtain a set of fusion enhanced infrared polarization images {F1,…, F n ,…, F N};

[0094] Step 3, the fused infrared polarization image is coded and reconstructed, and a guided filtered polarization image is obtained;

[0095] Step 3.1, F n is input into Encoder_I and Encoder_D respectively, and the n-th infrared intensity fusion encoding tensor EI n and the n-th polarization fusion encoding tensor ED n are obtained, and then EI n and ED n are input into Decoder for decoding, and the n-th reconstructed infrared intensity image RI n and the n-th reconstructed infrared polarization degree image RD n are obtained accordingly;

[0096] Step 3.2, the n-th infrared intensity image I n is taken as a guide image, and the n-th polarization degree image D n and the n-th polarization combination parameter image C n are guided filtered respectively, so as to obtain the n-th guided filtered polarization degree image GD n and the n-th polarization combination parameter image GC n ;

[0097] Step 4. Establish a loss function between the input infrared polarized image and the fusion image; in this embodiment, the infrared fusion loss contains three parts, which are multi-scale structural similarity loss, encoding reconstruction loss, and fusion loss based on polarized feature reconstruction. The encoding reconstruction loss can make the infrared polarized features before and after encoding consistent, and the fusion loss based on polarized feature reconstruction restores the original Stokes vector according to Malus law, and then calculates the polarized total light intensity loss before and after fusion. This method can avoid directly introducing the loss calculation of the degree of polarization, so as to further suppress the noise background interference of the degree of polarization image;

[0098] Step 4.1, constructing a multi-scale structural similarity loss using formula (5)

[0099]

[0100] In formula (5), MWSSIM(·) represents a multi-scale structural similarity loss function;

[0101] Step 4.2, constructing an encoding reconstruction loss using formula (6)

[0102]

[0103] In formula (6), MAE represents an average absolute error loss function;

[0104] Step 4.3, calculating a fusion loss based on polarized feature reconstruction through formula (7)

[0105]

[0106] In formula (7), and respectively represent the reconstructed 0°, 60° and 120° direction polarized images;

[0107] Step 4.4, constructing a total loss function L of infrared polarized image fusion enhancement using formula (8) n :

[0108]

[0109] In formula (8), λ1, λ2, λ3 are three weighting coefficients; in this embodiment, the three weighting coefficients λ1, λ2, λ3 are 1, 0.25 and 0.25 respectively;

[0110] Step 5, based on the infrared intensity image set {I1,...,I n ,...,I N}, the degree of polarization image set {D1,...,D n..., D N ..., A n ..., A N ..., C n ..., C N ..., C n The total loss function L is calculated by using a gradient descent algorithm to train the infrared polarization image fusion network, and the network parameters are updated, and when the number of training iterations reaches a set number of times, the training is stopped, thereby obtaining a trained infrared polarization image fusion enhancement model for realizing fusion and enhancement of infrared intensity and polarization degree images.

[0111] In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.

[0112] In this embodiment, a computer readable storage medium has a computer program stored thereon, and the computer program performs the steps of the above method when executed by a processor.

[0113] Table 1 Comparison of experimental results of different fusion methods

[0114] Metrics RFN-Nest DIDFuse U2Fusion PFNet SeAFusion SwinFusion TIPFNet Ours EN↑ 6.5921 5.985 5.7878 6.6813 7.0209 6.8844 6.4994 7.0884 SD↑ 68.5392 75.2079 57.6935 59.8444 76.7233 75.6023 57.7481 81.2445 CEIQ↑ 2.9194 2.4600 2.4907 3.0212 3.2302 3.1910 2.9547 3.2928 SSEQ↓ 52.971 43.0697 56.9430 44.1631 36.6161 41.0155 42.7571 29.2761 PIQE↓ 71.8132 54.59 55.9085 56.2109 49.9489 49.7075 51.6895 46.6389

[0115] In Table 1, Ours represents the infrared polarization image fusion enhancement method based on a learnable low-rank representation of the present application; RFN-Nest (Residual Fusion Network-Nest) is a residual fusion-based coding and decoding network, DIDFuse (Deep Image Decomposition Fuse) is a coding and decoding network based on image background and detail feature decomposition, U2Fusion (Unified and Unsupervised End-to-End Image Fusion Network) is an unsupervised end-to-end image fusion network, PFNet (Polarization Image Fusion Network) is a dense connection fusion network based on polarization images, SeAFusion (Semantic-Aware real-time Image Fusion Network) is a fusion method based on image fusion and segmentation linkage, SwinFusion (SwinTransformer-based Fusion) and TIPFNet (Transformer-based Infrared Polarization Image Fusion Network) are fusion methods based on Transformer. EN represents the information entropy fusion index, SD represents the image standard deviation, CEIQ (Contrast Enhance-based Image Quality) represents the image evaluation index based on contrast enhancement, SSEQ (Spatial-Spectral Entropy-based Quality) represents the evaluation index based on spatial and frequency domain information entropy, and PIQE (Perception-based Image Quality Evaluator) represents the evaluation index based on real-world visual perception. Table 2 is a comparison of model parameters of current lightweight fusion networks, wherein LRRNet (Low-Rank Representation Network) is a fusion network based on low-rank representation;

[0116] Table 2 Comparison of model parameters of lightweight fusion networks

[0117] Methods RFN-Nest SwinFusion U2Fusion LRRNet Ours parameters(M)↓ 7.524 0.974 0.659 0.049 0.039

[0118] The experimental results in Table 2 show that the present application has better overall performance than the above-mentioned image fusion methods under the current indicators, and the model has fewer network parameters and is more lightweight, meeting the real-time processing requirements.

Claims

1. An infrared polarization image fusion enhancement method based on a learnable low-rank representation, characterized in that, is performed as follows: Step 1, obtaining an infrared polarization image, and calculating a polarization Stokes vector; Obtain N sets of infrared polarization images containing 0°, 45°, 90°, and 135° directions respectively, and calculate the corresponding N sets of polarization Stokes vectors {{I1,Q1,U1}...,{I n Q n U n },...,{I N Q N U N }}, thus obtaining the infrared intensity image set {I1,...,I n ,...,I N }, Polarization degree image set {D1,...,D n ,...,D N }, Polarization angle image set {A1,...,A n ,...,A N } and the set of polarization combination parameter images {C1,...,C n ,...,C N }, where I n Q represents the first component of the nth infrared intensity image, i.e., the nth group of polarization Stokes vectors. n and U n These represent the second and third components of the nth polarization Stokes vector, respectively. This represents the nth polarization image. This represents the nth polarization angle image. This represents the nth polarization combination parameter image; Step 2, constructing an infrared polarization image fusion network based on a learnable low-rank representation, including: an infrared intensity low-rank feature encoding module Encoder_I, an infrared polarization low-rank feature encoding module Encoder_D, an attention guided filter fusion module AGFF, and a fusion feature decoding module Decoder; Step 2.1, the nth infrared intensity image I n After inputting into the Encoder_I for processing, the nth infrared intensity encoding feature tensor EN_I is obtained n ; Step 2.2, construct Encoder_D with the same structure as Encoder_I, and the nth polarization degree image D n After processing in the input Encoder_D, the nth polarization encoding feature tensor EN_D is obtained n ; Step 2.

3. Extract EN_I in channel dimension n The first half of the feature tensor, i.e., the infrared intensity low-rank encoding feature tensor, is denoted as L_I n Meanwhile, extract EN_D n The second half of the feature tensor, i.e., the polarization sparse encoding feature tensor, is denoted as S_D n ; Step 2.4, the attention guided filter fusion module AGFF fuses L_I n with S_D n to obtain the n-th fused feature tensor IG n ; Step 2.5, the fusion feature decoding module Decoder is composed of H feature decoding units DeU and a regression output unit RU; wherein each feature decoding unit DeU is composed of a convolution layer with a convolution kernel of 3x3 and a LeakyReLU activation layer, and the regression output unit RU is composed of a convolution layer with a convolution kernel of 3x3 and a Tanh activation layer; The n-th fused feature tensor IG n is input into a fusion feature decoding module Decoder for processing, and an n-th fusion enhanced infrared polarization image F n is output, so as to obtain a set of fusion enhanced infrared polarization images {F n , …, F N} Step 3, encoding and decoding the fused infrared polarization image to obtain a guided filtered polarization image; Step 3.1, input F n into Encoder_I, Encoder_D respectively, get the n-th infrared intensity fusion encoding tensor EI n and the n-th polarization fusion encoding tensor ED n , respectively. n and ED n are input into Decoder respectively for decoding, and the corresponding n-th reconstructed infrared intensity image RI n and the n-th reconstructed infrared polarization degree image RD n are obtained. Step 3.2, obtain the nth infrared intensity image I n As a guide image, the nth polarization degree image D n With the nth polarization combination parameter image C n Guided filtering is performed, thereby obtaining the nth guided filtered polarization degree image GD n With the nth polarization combination parameter image GC n ; Step 4, establishing a loss function between the input infrared polarization image and the fused image; Step 4.1, Constructing multi-scale structural similarity loss with formula (4) In formula (4), MWSSIM(·) represents a multi-scale structural similarity loss function; Step 4.2, Constructing the encoding reconstruction loss with formula (5) In formula (5), MAE represents an average absolute error loss function; Step 4.3, Compute the fusion loss based on the polarization feature reconstruction by formula (6) In formula (6), with respectively represent the reconstructed 0°, 60° and 120° directional polarized images of the nth image. Step 4.4, constructing the total loss function L for infrared polarization image fusion enhancement using formula (7) n : In formula (7), λ1, λ2, and λ3 are three weighted coefficients; Step 5, based on the infrared intensity image set {I1,...,I n ,...,I N}, the degree of polarization image set {D1,...,D n ,...,D N}, the polarization angle image set {A1,...,A n ,...,A N} and the polarization combination parameter image set {C1,...,C n ,...,C N}, the infrared polarization image fusion network is trained by using a gradient descent algorithm, and a total loss function L n is calculated to update the network parameters, and when the number of training iterations reaches a set number of times, the training is stopped, thereby obtaining a trained infrared polarization image fusion enhancement model for realizing fusion and enhancement of infrared intensity and polarization degree images. 2.The infrared polarized image fusion enhancement method based on a learnable low-rank representation according to claim 1, characterized in that, The infrared intensity low-rank feature encoding module Encoder_I in the step 2.1 is sequentially composed of a convolution layer with a convolution kernel of 3x3, a LeakyReLU activation layer, and M low-rank feature extraction modules, wherein the mth low-rank feature extraction module is denoted as LR-conv m , m ∈ [1, M]; the mth low-rank feature extraction module LR-conv m includes a convolution layer with a convolution kernel of 1x1, a BN layer, a PReLU activation layer, and two parallel branches in sequence, wherein the first branch is composed of a single convolution layer with a convolution kernel of 3x3, and the second branch is sequentially composed of L+1 low-rank feature representation blocks; wherein the lth low-rank feature representation block of the mth low-rank feature extraction module LR-conv m is denoted as l ∈ [0, L]; The nth infrared intensity image I n After being input into the Encoder_I and sequentially processed by a 3x3 convolution layer and a LeakyReLU activation layer, the nth infrared pre-encoding feature tensor IPinput is obtained n ; When m=1, the feature tensor IPinput n LR-conv, the m-th low-rank feature extraction module m The m-th input feature tensor is denoted as And through LR-conv m After processing with a 1×1 convolutional layer, a BN layer, and a PReLU activation layer, the m-th infrared convolutionally encoded feature tensor is obtained. After the processing of the first branch, the mth infrared base encoding feature tensor is obtained When l = 0, As the lth input feature tensor of , denoted as After being processed by a convolution layer with a convolution kernel of 1x1, the lth feature tensor is obtained After being processed by the activation function h θ (·) shown in formula (1), the lth activated feature tensor is output , denoted as In formula (1), sign(·) is a sign function, and θ is a threshold value; When l = 1, 2, ..., L, the (l-1)th low-rank feature block is represented. The (l-1)th output feature tensor As LLRR_Block l m The l-th input feature tensor And input Thus, by using equation (2), we can obtain The l-th output feature tensor Finally, the Lth low-rank feature block of the second branch is represented. Obtain the Lth output feature tensor In formula (2), conv1 represents a first convolution layer with a convolution kernel of 1×1, conv2 represents a second convolution layer with a convolution kernel of 1×1, λ R is a residual coefficient; The first branch output is Serially connected with After the processing of BN layer, PReLU activation layer and Channel Attention channel attention layer, the mth infrared encoded aggregated feature tensor is obtained Then, the mth feature tensor After residual connection, the mth infrared low-rank sparse feature tensor is output When m = 2, 3, …, M, the following is obtained Input the mth low-rank feature extraction module LR-conv m The input image tensor is processed in the module, and the output feature tensor is obtained Thus, the Mth low-rank feature extraction module LR-conv M Outputs the feature tensor And as the nth infrared intensity encoding feature tensor of Encoder_I, denoted as EN_I n . 3.The infrared polarized image fusion enhancement method based on a learnable low-rank representation according to claim 1, characterized in that, The attention guided filter fusion module AGFF is composed of an attention guided filter block AGF and a feature fusion block FFB, and the nth fused feature tensor IG is obtained according to the following process n ; Step 2.4.1, L_I n and S_D n are input into the attention guided filtering block AGF, first pass through the convolution kernel of 1*1 convolution layer processing, obtain two feature tensors and superimposed, then pass through ReLU activation layer, convolution kernel of 1*1 convolution layer and Sigmoid activation layer processing in turn, get the nth attention feature tensor T n , finally L_I n as a guide map, using T n obtain the filtering coefficient, used for guided filtering operation on S_D n , thereby obtaining the nth filtered feature tensor G n ; Step 2.4.2, L_I n is concatenated with G n , and then input into the feature fusion block FFB, and sequentially processed by the convolution layer with a convolution kernel of 3x3 and the Sigmoid activation layer, to obtain the feature tensor, which is segmented in the channel dimension. The first half of the feature tensor is denoted as FI n , and the second half of the feature tensor is denoted as FG n ; and thus the nth fused feature tensor IG n is obtained by formula (3): In formula (4), respectively, and L_I n , G n vector parameters to be learned with the same number of channels.

4. The method of claim 3, wherein the method further comprises: The guided filtering operation in step 2.4.1 is to obtain a local window centered at the pixel coordinate k using formula (8) Two linear coefficients of the local window According to The guided image L_I n is subjected to a local linear weighting operation to obtain the filtered feature tensor G n ; In formula (8), N k is the number of pixels in the local window n (i) represents the value of the i-th pixel point of the n-th attention feature tensor T n n (i) represents the value of the i-th pixel point of the n-th infrared intensity low-rank encoding feature tensor L_I n n (i) represents the value of the i-th pixel point of the n-th polarization sparse encoding feature tensor S_D n n (i) represents the value of the i-th pixel point of the n-th normalized attention feature tensor X n .​​​​ 5. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store a program supporting the processor to execute the infrared polarization image fusion enhancement method of any one of claims 1-4, and the processor is configured to execute the program stored in the memory.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the infrared polarization image fusion enhancement method of any one of claims 1-4.

Images