Misaligned infrared and visible image fusion method based on information compensation and prediction correction

Abstract

The present invention relates to a method for fusion of misaligned infrared-visible images based on information compensation and predictive correction. The method comprises the following steps: inputting a fixed image and an offset image into a feature encoder; inputting the fixed and offset features into a cross-modal information compensation module to transform unimodal features into omnimodal features, thereby reducing the modal difference between the fixed and offset features; inputting the fixed and offset features after the modal difference is reduced into a dual deformation field predictive correction module to correct the offset features; and inputting the corrected features and the fixed features into a fusion module to fuse complementary information and reconstruct an infrared-visible fused image. The method can correct spatial misalignment of infrared-visible image pairs and eliminate artifacts and offsets in the fusion result, thereby improving fusion performance.

Description

Technical Field

[0001] The invention relates to a misaligned infrared and visible image fusion method based on information compensation and prediction correction, and belongs to the technical field of image fusion. Background Art

[0002] Infrared and visible image fusion integrates information from both infrared and visible light images to produce a more informative fused image. Infrared images capture thermal information but lack the ability to describe texture details. Visible light images, on the other hand, capture surface texture details but can lose significant information when affected by extreme weather, occlusion, and illumination. Therefore, integrating information from these two modalities results in a fused image that preserves both the texture details in the visible light image and the thermal information in the infrared image. However, in practical applications, spatial misalignment of infrared and visible light image pairs is common due to sensors capturing images in different environments. Directly fusing misaligned infrared and visible light image pairs often produces a fused image riddled with artifacts and offsets, severely impacting the performance of downstream tasks such as image recognition, semantic segmentation, and object tracking, significantly limiting the application of infrared and visible light image fusion in real-world scenarios. To address this dilemma, a misaligned infrared and visible image fusion method is proposed that incorporates cross-modal information compensation and dual deformation field prediction correction. This method mitigates the negative impact of misaligned source image pairs on the fusion result, resulting in a visually appealing fusion result free of artifacts and offsets. Summary of the Invention

[0003] In order to address the shortcomings of existing methods, the present invention provides a misaligned infrared-visible image fusion method based on information compensation and predictive correction. The present invention can correct the spatial misalignment of infrared-visible light image pairs and eliminate artifacts and offsets in the fusion results, thereby improving the fusion performance.

[0004] The technical solution of the present invention is: a misaligned infrared-visible image fusion method based on information compensation and prediction correction, the method comprising:

[0005] Step 1: Obtain a training dataset for unaligned infrared-visible image fusion, where the fixed image is a visible light image and the offset image is an infrared image;

[0006] Step 2: Input the fixed image and the offset image into the feature encoder to obtain fixed features and offset features;

[0007] Step 3: Input the fixed features and the offset features into the cross-modal information compensation module, and convert the two single-modal features into full-modal fixed features and full-modal offset features, respectively, to reduce the modal difference between the fixed features and the offset features;

[0008] Step 4: Input the fixed features and offset features after the modal difference is reduced into the dual deformation field prediction and correction module, correct the offset features, and obtain the corrected features;

[0009] Step 5: Input the correction features and fixed features into the feature fusion module and reconstruct the infrared-visible light fusion image.

[0010] As a further solution of the present invention, in step 1, each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image; the size of the image of each modality is 256×256;

[0011] For pre-aligned infrared and visible light image pairs Apply the same offset to obtain another pair of aligned offset infrared-visible image pairs ,Pick As an unaligned infrared-visible image pair, the fixed image is a visible light image , the migration image is the infrared image ; Preprocess the training dataset for unaligned infrared-visible image fusion. The preprocessing includes: random flipping, random rotation and offset of the data, and normalizing the processed images.

[0012] As a further solution of the present invention, step 2 includes:

[0013] Fixed image is visible light image , the migration image is the infrared image , the visible light image and infrared images Input fixed feature encoder respectively and offset feature encoder In the fixed feature and offset features Fixed features and offset features The generation process is expressed as:

[0014] ;

[0015] Fixed feature encoder and offset feature encoder The feature extraction blocks are composed of four 3×3 convolutional layers and ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

[0016] As a further solution of the present invention, in step 3, the specific operation of the cross-modal information compensation module is as follows:

[0017] Fixed image is visible light image , the migration image is the infrared image , the unaligned infrared images and visible light images After CLIP, the features are obtained and , and use the broadcast mechanism to and Expand to and , so that it is consistent with and have the same size; and , fixed features and offset features After cross-modal compensation and 3×3 convolutional layer processing, the full-modal fixed features are obtained and full modal offset characteristics :

[0018] .

[0019] As a further solution of the present invention, step 4 includes:

[0020] Full modal fixed features after reducing modal differences and full modal offset characteristics , and the visible light image gradient of the image to be aligned and infrared image gradient Generate deformation fields separately and ; Deformation field and The generation process of is expressed by the following formula:

[0021] ϕ f = DFPNet1 [ F vis,c , F ˜ ir,c ] ϕ g = DFPNet2 [ ∇ I vis , ∇ I ˜ ir ] ;

[0022] Among them, DFPNet1 and DFPNet2 are U-Net-like networks with the same structure but different parameters;

[0023] The deformation field and Applied to infrared images On the rectified image, the gradient of the rectified image is compared with Gradient The pixels are spliced and global average pooling and global maximum pooling are performed on the channels and spatial positions respectively. The pooled results are added and sent to the Sigmoid activation function to obtain the correction weight values of all pixels on the channels and spatial positions. Through the broadcast mechanism, the weight values on the channels and the weight values on the spatial positions are formed into a matrix that can be multiplied. After the dot multiplication operation of the matrix, the weight matrix for correcting and integrating the deformation field is obtained. and ; The corrected deformation field after adjustment and integration Expressed as:

[0024] ;

[0025] Correcting the deformation field With offset feature After the Warp operation, the correction features are obtained .

[0026] As a further solution of the present invention, in step 5, the processing flow of the feature fusion module includes:

[0027] Correction Features and fixed features Through the Sigmoid activation function, the weight map is obtained and ; The weight map and With calibration characteristics and fixed features The features are concatenated and further extracted through a 3×3 convolutional layer. The global average pooling and global maximum pooling operations are jointly used for the features on each channel, and the pooled results are added and sent to the multi-layer perceptron. Finally, the weights that can express the importance of each channel are obtained after Sigmoid activation function processing. and The outputs of the infrared and visible light branches after global maximum pooling are added together, and the result of the addition is sent to the multi-layer perceptron. Then, the weights of the infrared image and the visible light image at the same channel position are obtained through the Sigmoid activation function. ; weight The original weights added to the infrared and visible branches respectively and Thus, the final weights of the two branches are obtained and :

[0028] ;

[0029] according to and With calibration characteristics and fixed features , and finally get the fusion feature ; Fusion features Expressed as:

[0030] ;

[0031] in and are respectively and Broadcasted, Express matrix element-by-element multiplication;

[0032] The fusion features Feed it into the decoder which consists of a 3×3 convolutional layer and Tanh activation function , and get the fused image .

[0033] As a further solution of the present invention, step 3 further includes:

[0034] The consistency loss is defined To optimize the 3×3 convolution layer, consistency loss Expressed as:

[0035] ;

[0036] in, express norm, To fusion features, is a full-modal fixed feature, is the full modal offset feature;

[0037] and It is obtained by learning the full-modal feature representation; the specific operations are as follows:

[0038] Align these two pairs of infrared and visible light images and Input to the parameter-shared feature encoder and The extracted features are sent to the feature fusion module in step 5 for fusion, and finally the fused full-modal fusion features are obtained. and .

[0039] As a further solution of the present invention, step 5 further includes:

[0040] Use texture loss to preserve the texture details of infrared and visible light images; texture loss The calculation formula is as follows:

[0041] ;

[0042] in, 、 、 Represent the fused images , infrared images , visible light images The gradient, Indicates pixel-by-pixel maximum value selection;

[0043] In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level. is defined as follows:

[0044] ;

[0045] ;

[0046] in, and Infrared images and visible light images The significance matrix of and Infrared images and visible light images The weight graph of .

[0047] The present invention also provides a misaligned infrared-visible image fusion system based on information compensation and prediction correction. The system includes: a module for executing the misaligned infrared-visible image fusion method based on information compensation and prediction correction.

[0048] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for fusion of misaligned infrared and visible images based on information compensation and prediction correction is implemented.

[0049] The beneficial effects of the present invention are:

[0050] 1. This invention converts single modal features into full modal features through a cross-modal information compensation mechanism, effectively reducing modal differences and thus alleviating the challenges posed by modal differences to cross-modal feature matching;

[0051] 2. This invention uses a dual deformation field prediction and correction approach to predict two deformation fields from the perspectives of features and source image gradients, and uses standard gradients to perceive the accuracy of deformation field predictions. This integration generates a more accurate deformation field, thereby improving the accuracy of feature correction.

[0052] 3. This invention designs an adaptive perceptual feature fusion module that senses the importance of each feature to the fusion result and adaptively fuses the features based on these perceived scores, reconstructing a high-quality fused image free of artifacts and offsets. This addresses the difficulty existing methods face in fusing misaligned infrared and visible light image pairs.

[0053] 4. A large number of experimental results on public datasets show that the method proposed in this paper can effectively fuse unaligned infrared and visible light image pairs, and its performance is better than existing advanced methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of the process structure of the present invention;

[0055] Figure 2 Schematic diagram of a cross-modal information compensation module of the present invention;

[0056] Figure 3 Schematic diagram of the dual deformation field prediction and correction module of the present invention;

[0057] Figure 4 Schematic diagram of the adaptive perception feature fusion module of the present invention;

[0058] Figure 5 This is a comparison chart of the experimental results of the method of the present invention and the existing method;

[0059] Figure 6 This is a quantitative comparison chart of the fusion results of the existing method and the method of the present invention. DETAILED DESCRIPTION

[0060] Example 1: Figures 1-6 As shown in the figure, the present invention addresses the spatial misalignment of infrared-visible image pairs, which introduces artifacts and offsets in the fusion results, thereby seriously affecting the performance of downstream tasks and limiting their practical applications. A misaligned infrared-visible image fusion method based on cross-modal information compensation and dual deformation field prediction correction is proposed.

[0061] A misaligned infrared-visible image fusion method based on information compensation and prediction correction, the method comprising:

[0062] Step 1: Obtain a training dataset for unaligned infrared-visible image fusion, where the fixed image is a visible light image and the offset image is an infrared image;

[0063] As a further solution of the present invention, in step 1, each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image; the size of the image of each modality is 256×256;

[0064] For pre-aligned infrared and visible light image pairs Apply the same offset to obtain another pair of aligned offset infrared-visible image pairs ,Pick As an unaligned infrared-visible image pair, the fixed image is a visible light image , the migration image is the infrared image ; Preprocess the training dataset for unaligned infrared-visible image fusion. The preprocessing includes: random flipping, random rotation and offset of the data, and normalizing the processed images.

[0065] Step 2: Input the fixed image and the offset image into the feature encoder to obtain fixed features and offset features;

[0066] As a further solution of the present invention, step 2 includes:

[0067] Fixed image is visible light image , the migration image is the infrared image , the visible light image and infrared images Input fixed feature encoder respectively and offset feature encoder In the fixed feature and offset features Fixed features and offset features The generation process is expressed as:

[0068] ;

[0069] Fixed feature encoder and offset feature encoder The feature extraction blocks are composed of four 3×3 convolutional layers and ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

[0070] Step 3: Input the fixed features and the offset features into the cross-modal information compensation module, and convert the two single-modal features into full-modal fixed features and full-modal offset features, respectively, to reduce the modal difference between the fixed features and the offset features;

[0071] As a further solution of the present invention, in step 3, the schematic diagram of the cross-modal information compensation module is as follows: Figure 2As shown in Figure 2, the specific operations of the cross-modal information compensation module are as follows:

[0072] Fixed image is visible light image , the migration image is the infrared image , the unaligned infrared images and visible light images After CLIP, the features are obtained and , and use the broadcast mechanism to and Expand to and , so that it is consistent with and have the same size; and , fixed features and offset features After cross-modal compensation and 3×3 convolutional layer processing, the full-modal fixed features are obtained and full modal offset characteristics :

[0073] .

[0074] Step 4: Input the fixed features and offset features after the modal difference is reduced into the dual deformation field prediction and correction module, correct the offset features, and obtain the corrected features; Figure 3 As shown, step 4 includes:

[0075] Full modal fixed features after reducing modal differences and full modal offset characteristics , and the visible light image gradient of the image to be aligned and infrared image gradient Generate deformation fields separately and ; Deformation field and The generation process of is expressed by the following formula:

[0076] ϕ f = DFPNet1 [ F vis,c , F ˜ ir,c ] ϕ g = DFPNet2 [ ∇ I vis , ∇ I ˜ ir ] ;

[0077] Among them, DFPNet1 and DFPNet2 are U-Net-like networks with the same structure but different parameters;

[0078] The deformation field and Applied to infrared images On the rectified image, the gradient of the rectified image is compared with Gradient The pixels are spliced and global average pooling and global maximum pooling are performed on the channels and spatial positions respectively. The pooled results are added and sent to the Sigmoid activation function to obtain the correction weight values of all pixels on the channels and spatial positions. Through the broadcast mechanism, the weight values on the channels and the weight values on the spatial positions are formed into a matrix that can be multiplied. After the dot multiplication operation of the matrix, the weight matrix for correcting and integrating the deformation field is obtained. and ; The corrected deformation field after adjustment and integration Expressed as:

[0079] ;

[0080] Correcting the deformation field With offset feature After the Warp operation, the correction features are obtained .

[0081] Step 5: Input the correction features and fixed features into the feature fusion module and reconstruct the infrared-visible light fusion image.

[0082] As a further solution of the present invention, in step 5, Figure 4 As shown in Figure 2, the processing flow of the feature fusion module includes:

[0083] Correction Features and fixed features Through the Sigmoid activation function, the weight map is obtained and ; The weight map and With calibration characteristics and fixed features The features are concatenated and further extracted through a 3×3 convolutional layer. The global average pooling and global maximum pooling operations are jointly used for the features on each channel, and the pooled results are added and sent to the multi-layer perceptron. Finally, the weights that can express the importance of each channel are obtained after Sigmoid activation function processing. and The outputs of the infrared and visible light branches after global maximum pooling are added together, and the result of the addition is sent to the multi-layer perceptron. Then, the weights of the infrared image and the visible light image at the same channel position are obtained through the Sigmoid activation function. ; weight The original weights added to the infrared and visible branches respectively and Thus, the final weights of the two branches are obtained and :

[0084] ;

[0085] according to and With calibration characteristics and fixed features , and finally get the fusion feature ; Fusion features Expressed as:

[0086] ;

[0087] in and are respectively and Broadcasted, Express matrix element-by-element multiplication;

[0088] The fusion features Feed it into the decoder which consists of a 3×3 convolutional layer and Tanh activation function , and get the fused image .

[0089] As a further solution of the present invention, step 3 further includes:

[0090] In order to make and Can effectively eliminate the impact of modal differences and prevent and The complementary information is lost in the , and the consistency loss is defined To optimize the 3×3 convolution layer, consistency loss Expressed as:

[0091] ;

[0092] in, express norm, To fusion features, is a full-modal fixed feature, is the full modal offset feature;

[0093] and It is obtained by learning the full-modal feature representation; the specific operations are as follows:

[0094] Align these two pairs of infrared and visible light images and Input to the parameter-shared feature encoder and The extracted features are sent to the feature fusion module in step 5 for fusion, and finally the fused full-modal fusion features are obtained. and .

[0095] As a further solution of the present invention, step 5 further includes:

[0096] Use texture loss to preserve the texture details of infrared and visible light images; texture loss The calculation formula is as follows:

[0097] ;

[0098] in, 、 、 Represent the fused images , infrared images , visible light images The gradient, Indicates pixel-by-pixel maximum value selection;

[0099] In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level. is defined as follows:

[0100] ;

[0101] ;

[0102] in, and Infrared images and visible light images The significance matrix of and Infrared images and visible light images In addition, in order to mine the complementary information of the input multimodal features as much as possible, the complementary information mining loss is defined :

[0103] ;

[0104] The present invention also provides a misaligned infrared-visible image fusion system based on information compensation and prediction correction. The system includes: a module for executing the misaligned infrared-visible image fusion method based on information compensation and prediction correction.

[0105] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for fusion of misaligned infrared and visible images based on information compensation and prediction correction is implemented.

[0106] To verify the effectiveness of the method of the present invention, the performance of the proposed method was evaluated on the public RoadScene, MSRS, M3FD, and CVC-14 datasets. In this process, the corresponding model of the method of the present invention was trained on the training set of the RoadScene dataset, and the results were tested on the test sets of the RoadScene, MSRS, M3FD, and CVC-14 datasets. The training set consists of 221 pairs of infrared and visible light image pairs from the RoadScene dataset; the test set consists of 20 pairs of images from RoadScene, 20 pairs of images from MSRS, 17 pairs of images from M3FD, and 21 pairs of images from the real scene dataset CVC-14. The algorithm of the present invention was developed under the Pytorch1.10.1 framework and trained on a single NVIDIA GTX3090 graphics card (24G video memory). During training, we used the Adam optimizer to optimize the model parameters. In this process, the batch size was set to 8. The learning rate decay strategy of “CosineAnnealing” was adopted, the initial learning rate was 0.001, and a total of 1000 rounds of training were performed.

[0107] Furthermore, the present invention is compared with IMF, IVFWSR, MURF, RFVIF, SemLA, SuperFusion and UMF-CMGR in terms of the visual effects of the fusion results for misaligned infrared and visible light images. Figure 5 As shown. Figure 5 It can be seen that the method proposed in the present invention can better correct spatial misalignment and obtain a high-fidelity fused image with reduced artifacts and offsets.

[0108] Six commonly used image quality evaluation indicators are selected to objectively evaluate the quality of the fusion results, including the correlation coefficient ( ), gradient-based fusion performance metrics ( ), Chen-Blum indicator( ), Chen-Varshney index ( ), the sum of linearly independent correlations ( ), and structural similarity ( ). The degree of linear correlation between the fused image and the source image is evaluated to measure their similarity. The amount of edge information transferred from the source image to the fused image is evaluated. The similarity between the main features of the fused image and the source image is measured based on the human visual system. It considers both edge information in the fused image and human visual perception. The difference map is used to evaluate the degree of correlation between the fused image and the source image. By comparing the fused image with the source image, the information loss and distortion of the fused image are quantified. Among these evaluation indicators, The lower the value of , the better the quality of the fused image; the higher the values of the other indicators, the better the fusion quality.

[0109] Figure 6 This is a quantitative comparison chart of the fusion results of the existing method and the method of the present invention. The x-axis represents each method, and the y-axis represents the index value. Figure 6 The y-axis in the figure has no dimension or unit and is a constant. The up and down arrows after the indicator represent that the larger / smaller the indicator value is, the better the fused image quality is. The upper and lower boundaries of each box represent the upper and lower quartiles respectively. The black line represents the median and the red line represents the mean. Figure 6 As can be seen, our method achieves the best performance on four metrics and suboptimal performance on two metrics. However, the upper and lower bounds of our method on these two suboptimal metrics are smaller, indicating that our method is more robust than other methods.

[0110] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A misaligned infrared and visible image fusion method based on information compensation and prediction correction, characterized by: The method comprises: Step 1: Obtain a training dataset for unaligned infrared-visible image fusion, where the fixed image is a visible light image and the offset image is an infrared image; Step 2: Input the fixed image and the offset image into the feature encoder to obtain fixed features and offset features; Step 3: Input the fixed features and the offset features into the cross-modal information compensation module, and convert the two single-modal features into full-modal fixed features and full-modal offset features, respectively, to reduce the modal difference between the fixed features and the offset features; Step 4: Input the fixed features and offset features after the modal difference is reduced into the dual deformation field prediction and correction module, correct the offset features, and obtain the corrected features; Step 5: Input the correction features and fixed features into the feature fusion module and reconstruct the infrared-visible light fusion image; In step 3, the specific operations of the cross-modal information compensation module are as follows: Fixed image is visible light image , the migration image is the infrared image , the unaligned infrared images and visible light images After CLIP, the features are obtained and , and use the broadcast mechanism to and Expand to and , so that it is consistent with and have the same size; and , fixed features and offset features After cross-modal compensation and 3×3 convolutional layer processing, the full-modal fixed features are obtained and full modal offset characteristics : ； The step 4 comprises: Full modal fixed features after reducing modal differences and full modal offset characteristics , and the visible light image gradient of the image to be aligned and infrared image gradient Generate deformation fields separately and ; Deformation field and The generation process of is expressed by the following formula: ； Among them, DFPNet1 and DFPNet2 are U-Net-like networks with the same structure but different parameters; The deformation field and Applied to infrared images On the rectified image, the gradient of the rectified image is compared with Gradient The pixels are spliced and global average pooling and global maximum pooling are performed on the channels and spatial positions respectively. The pooled results are added and sent to the Sigmoid activation function to obtain the correction weight values of all pixels on the channels and spatial positions. Through the broadcast mechanism, the weight values on the channels and the weight values on the spatial positions are formed into a matrix that can be multiplied. After the dot multiplication operation of the matrix, the weight matrix for correcting and integrating the deformation field is obtained. and ; The corrected deformation field after adjustment and integration Expressed as: ； Correcting the deformation field With offset feature After the Warp operation, the correction features are obtained .

2. The misaligned infrared-visible image fusion method based on information compensation and prediction correction according to claim 1, characterized in that: In step 1, each infrared-visible light image pair in the training data set consists of an infrared image and a visible light image; the size of each modality image is 256×256; For pre-aligned infrared and visible light image pairs Apply the same offset to obtain another pair of aligned offset infrared-visible image pairs ,Pick As an unaligned infrared-visible image pair, the fixed image is a visible light image , the migration image is the infrared image ; Preprocessing includes: random flipping, random rotation and offset of the data, and normalizing the processed images.

3. The misaligned infrared-visible image fusion method based on information compensation and prediction correction according to claim 1, characterized in that: The step 2 includes: Fixed image is visible light image , the migration image is the infrared image , the visible light image and infrared images Input fixed feature encoder respectively and offset feature encoder In the fixed feature and offset features Fixed features and offset features The generation process is expressed as: ； Fixed feature encoder and offset feature encoder The feature extraction blocks are composed of four 3×3 convolutional layers and ReLU activation functions connected by dense connections, and finally composed of a Concat operation and a 1×1 convolutional layer.

4. The misaligned infrared-visible image fusion method based on information compensation and prediction correction according to claim 1, characterized in that: In step 5, the processing flow of the feature fusion module includes: Correction Features and fixed features Through the Sigmoid activation function, the weight map is obtained and ; The weight map and With calibration characteristics and fixed features The features are concatenated and further extracted through a 3×3 convolutional layer. The global average pooling and global maximum pooling operations are jointly used for the features on each channel, and the pooled results are added and sent to the multi-layer perceptron. Finally, the weights that can express the importance of each channel are obtained after Sigmoid activation function processing. and The outputs of the infrared and visible light branches after global maximum pooling are added together, and the result of the addition is sent to the multi-layer perceptron. Then, the weights of the infrared image and the visible light image at the same channel position are obtained through the Sigmoid activation function. ; weight The original weights added to the infrared and visible branches respectively and Thus, the final weights of the two branches are obtained and : ； according to and With calibration characteristics and fixed features , and finally get the fusion feature ; Fusion features Expressed as: ； in and are respectively and Broadcasted, Express matrix element-by-element multiplication; The fusion features Feed it into the decoder which consists of a 3×3 convolutional layer and Tanh activation function , and get the fused image .

5. The misaligned infrared-visible image fusion method based on information compensation and prediction correction according to claim 1, characterized in that: The step 3 further comprises: The consistency loss is defined To optimize the 3×3 convolution layer, consistency loss Expressed as: ； in, express norm, To fusion features, is a full-modal fixed feature, is the full modal offset feature; and It is obtained by learning the full-modal feature representation; the specific operations are as follows: Align these two pairs of infrared and visible light images and Input to the parameter-shared feature encoder and The extracted features are sent to the feature fusion module in step 5 for fusion, and finally the fused full-modal fusion features are obtained. and .

6. The misaligned infrared-visible image fusion method based on information compensation and prediction correction according to claim 4, characterized in that: The step 5 further comprises: Use texture loss to preserve the texture details of infrared and visible light images; texture loss The calculation formula is as follows: ； in, 、 、 Represent the fused images , infrared images , visible light images The gradient, Indicates pixel-by-pixel maximum value selection; In terms of preserving image content information, content loss is defined to bring the fused image closer to the source image at the pixel level. is defined as follows: ； ； in, and Infrared images and visible light images The significance matrix of and Infrared images and visible light images The weight graph of .

7. Misaligned infrared and visible image fusion system based on information compensation and prediction correction, characterized in that: The system includes: a module for executing the misaligned infrared-visible image fusion method based on information compensation and prediction correction according to any one of claims 1 to 6.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the misaligned infrared-visible image fusion method based on information compensation and prediction correction as described in any one of claims 1 to 6 is implemented.

Images