Unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment

Abstract

This invention relates to a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment, belonging to the field of computer vision and image processing technology. The invention includes: constructing a feature encoder to extract multi-scale features from infrared and visible light images and establish a feature candidate library; inputting the unregistered infrared and visible light features into a multi-view embedding module to predict multiple pseudo-deformation fields; applying the pseudo-deformation fields to the infrared features for progressive cross-scale alignment; inputting the aligned infrared and visible light features into a feature fusion module to extract fused features; inputting the fused features into a reconstruction module to generate the final fused image; training the feature encoder, the feature fusion convolution module, and the image reconstruction and thinning module; and using the trained network for the fusion processing of infrared and visible light images. This invention improves the fusion quality of infrared and visible light images in unaligned scenarios, enhancing the robustness of feature alignment and the clarity of image details.

Description

Technical Field

[0001] This invention relates to a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment, belonging to the field of computer vision and image processing technology. Background Technology

[0002] With the widespread application of infrared and visible light imaging technologies, infrared-visible light image fusion has become a key means to improve image perception quality and environmental adaptability. By fusing thermal radiation information from infrared images with rich texture details from visible light images, more complete, clear, and robust visual information can be obtained in various complex environments. However, in practical applications, due to differences in imaging equipment, viewing angle shifts, optical distortions, and other factors, infrared and visible light images often have registration errors, leading to poor or ineffective results from traditional fusion methods based on registration assumptions. Existing fusion methods typically rely on the assumption of strict registration, making it difficult to achieve high-precision fusion in cases of local displacement and overall pixel elastic deformation. Especially under unregistered conditions, direct image fusion methods are prone to introducing artifacts, blurring, and pixel structure misalignment, severely affecting the visual quality of the fused image and the accuracy of subsequent tasks (such as detection and recognition).

[0003] Specifically, this invention proposes a method for fusing unregistered infrared and visible light images based on multi-view embedding and progressive multi-scale alignment, and innovatively introduces a multi-view-multi-scale feature-level pseudo-deformation field prediction method. Summary of the Invention

[0004] The technical problem solved by this invention is that it provides a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment to overcome the problems of insufficient alignment accuracy and degraded fusion quality in the fusion of unregistered infrared and visible light images. The method of this invention effectively improves the consistency of cross-modal features and the quality of the fused image through multi-view pseudo-deformation field prediction and multi-scale progressive feature alignment strategy, and can achieve high-precision fusion without strict registration.

[0005] The technical solution of this invention is: a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment, the method comprising:

[0006] Step 1: Obtain infrared and visible light image pairs, and preprocess the input images to unify image size and normalize them;

[0007] Step 2: Construct a feature encoder, which includes a convolutional neural network branch and a Transformer branch. The feature encoder extracts local detail features and global context features from infrared and visible light images, respectively. The extracted features are then concatenated and fused to establish a feature candidate library.

[0008] Step 3: Construct a multi-view embedding module. Input the infrared and visible light features from the feature candidate library into the multi-view embedding module. Through correlation calculation and feature interaction, predict the pseudo-deformation fields of the first, second and third views respectively.

[0009] Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction in order from high to low levels, and gradually optimize the feature alignment effect at each scale.

[0010] Step 5: Construct a feature fusion module. Input the progressively aligned infrared and visible light features into the feature fusion convolution module. Perform feature fusion through convolution operations to generate a fused feature map. Input the fused feature map into the feature reconstruction and thinning module. Use convolution and activation operations to generate the final fused image.

[0011] Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module.

[0012] Step 7: Input the unregistered infrared and visible light images to be processed into the trained network, and output the high-quality fused image after alignment and fusion.

[0013] Further, Step 1 includes:

[0014] The preprocessing includes two stages. In the first stage, the infrared and visible light images are randomly cropped to a size of 128×128 and normalized before being input into the feature encoder, feature fusion module, and multi-view embedding module. In the second stage, the infrared and visible light images are scaled to a size of 400×304, and an affine transformation is applied to the infrared images, including different degrees of rotation and translation. Then, an elastic transformation is applied with a Gaussian function consisting of a kernel size of 63×63 and a standard deviation of 32 to increase the diversity of training samples while simulating the offset in real-world scenarios.

[0015] Furthermore, Step 2 includes:

[0016] Step 2.1: For feature extraction operations in the CNN and Restormer branches, the input infrared and visible light images are respectively input into the two branches of the feature encoder E for processing to obtain preliminary extracted features, specifically including:

[0017] First, input the registered infrared image and the visible light image pair (I ir ,I vi) and unregistered infrared images versus visible light images The input is fed into the Shallow Feature Extractor (SFE) to extract initial shallow features;

[0018] Next, the shallow features are input into the CNN branch and the Restormer branch respectively for deep feature extraction;

[0019] Then, multi-scale feature representations of infrared and visible light images are obtained from the CNN branch and the Restormer branch, respectively.

[0020] Finally, the obtained features are denoted as follows: and

[0021] Among them, I ir Indicates a strictly registered infrared image, I vi Represents a visible light image. This represents an misaligned infrared image, E represents the feature encoder, SFE represents the shallow feature extractor, the CNN branch uses the UIB Block structure, and the Restormer branch uses the Restormer module. and These represent the features extracted from different inputs through the CNN branch and the Restormer branch, respectively.

[0022] Step 2.2: For feature fusion operations within the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representations; specifically including:

[0023] First, the features extracted from the infrared image and the visible light image in their respective modalities are stitched together.

[0024] Next, the concatenated features are further fused and adjusted using two convolutional blocks (CBs).

[0025] Then, the aligned and unaligned infrared and visible light features after fusion are denoted as F, respectively. ir,f , and F vi,f This is used for subsequent generalized deformation field prediction and fusion tasks;

[0026] Where CB represents the convolution block operation, F ir,f This indicates the fused infrared signature. F represents the fused unaligned infrared features. vi,f This indicates the visible light characteristics after fusion;

[0027] Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the effectiveness of the features; this includes:

[0028] The fused infrared feature F ir,f and the fused visible light feature F vi,f The images are input into the reconstruction module RM to reconstruct the corresponding source infrared images I. ir,c Visible light image I vi,c .

[0029] Furthermore, in Step 3, the operation process of the multi-view embedding module includes:

[0030] Unaligned infrared features fused from the feature candidate library With the fused visible light feature F vi,f The input is fed into a multi-scale feature extraction network to extract features from the final scale layer. and

[0031] First, the features of the final scale layer and The correlation within the local neighborhood is calculated to obtain the local correlation matrix R. Δi,Δj (i,j), local correlation matrix R Δi,Δj (i,j) is defined as:

[0032]

[0033] Where Δi,Δj∈{-1,0,1} represents the pixel displacement amplitude within the local range; i and j represent the i-th row and j-th column of the feature map, respectively; C is the total number of channels of the feature, and k is the channel index, corresponding to the k-th channel among the C channels;

[0034] Next, the local correlation matrix R of all locations is... Δi,Δj (i,j) are concatenated to form the complete correlation matrix R, defined as:

[0035] R = [R -1,-1 ,R -1,0 ,R -1,1 ,...,R 1,1 ]

[0036] Then, the complete correlation matrix R is compared with the features of the final scale layer. Feature concatenation is performed, and the concatenated features are fed into a 3×3 convolutional layer and a Restormer module for further feature extraction. Then, the extracted features are input into three consecutive branches of the pseudo-deformation field prediction network (GDF): the first branch directly feeds the extracted features into the GDF for processing, predicting the pseudo-deformation field from the first-viewpoint. The extracted features are then fed into the second branch, the UIB_Block module, where they are combined with channel attention to further extract and enhance features. Subsequently, a second-view pseudo-deformation field was obtained through another GDF prediction. The third branch stitches together the pseudo-deformation fields from the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo-deformation field from the third perspective.

[0037] In the above operation, the GDF network consists of a 3×3 convolutional block with custom weights. It predicts the pseudo-deformation field from three different perspectives at the Nth scale. The enhanced features obtained at the Nth scale will be passed to the next scale to enhance information transmission and feature fusion.

[0038] Furthermore, in Step 4, the operation process of the progressive multi-scale alignment module includes:

[0039] First, the unaligned infrared features from the (N-1)th layer output by the multi-scale feature extraction network are... The pseudo-deformation field predicted using the previous scale For the registration of this misaligned feature, feature correction is performed sequentially from the highest layer to the lowest layer according to the scale number n = (N, N-1, ..., 1);

[0040] Next, at each scale, the pseudo-deformation field calculated based on the previous scale is... Unaligned infrared features of layer N-1 Perform progressive sampling transformation to obtain aligned infrared features.

[0041] Then, the aligned infrared features Input into the Restormer module at the current scale, and the visible light features at the same scale. Feature fusion is performed to improve the consistency of aligned features; the next step is to perform alignment of infrared features. Enhanced features compared to the previous scale The data is then fed into the second Restormer module for feature fusion to further enhance the information transfer between features at different scales. Step 3 is repeated to obtain the pseudo-deformation fields from three different perspectives at the corresponding scales.

[0042] Finally, feature correction and fusion at all scales are performed sequentially to progressively optimize the alignment of infrared and visible light features across multiple scales, providing a precise feature foundation for subsequent fusion tasks. This progressive multi-scale feature alignment process yields the final aligned infrared features. The process is defined as follows:

[0043]

[0044] Where n represents the scale number and N represents the total number of scale layers. This represents the unaligned infrared feature of the nth layer. This represents the pseudo-deformation field predicted at the nth layer, where W is the Warp operation, which involves sampling and transforming the pseudo-deformation field. This represents the visible light characteristics of the nth layer. This represents the aligned infrared features after applying a pseudo-deformation field transformation. It is the final aligned infrared feature.

[0045] Furthermore, in Step 5, the operation process of the feature fusion module includes:

[0046] First, the fused infrared features F that are aligned with each other are... ir,f With the fused visible light feature F vi,f The input is fed into the Feature Fusion Convolutional Module (FFCM), where features are fused through convolution operations to generate a fused feature map.

[0047] Next, the fused feature map is input into the Feature Reconstruction and Refinement (FRRB) module. In the FRRB module, the feature representation capability is first enhanced by the Channel Attention (CA) module. Then, features are further extracted using a convolutional block containing a 3×3 convolutional layer, a batch normalization (BN) layer, and a Leaky ReLU activation function. Finally, a 1×1 convolutional layer followed by a Sigmoid activation function is used to output the final fused image I. f The feature fusion and reconstruction process is defined as follows:

[0048] I f =FRRB(FFCM(F ir,f ,F vi,f )),

[0049] Wherein, FFCM represents the feature fusion convolution module, FRRB represents the feature reconstruction and refinement module, and I f This represents the final merged image.

[0050] Furthermore, Step 6 includes:

[0051] Step 6.1: The first stage involves training the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including:

[0052] First, the registered infrared image is paired with the visible light image (I ir ,I vi The fused infrared features F are input into the feature encoder E and extracted respectively. ir,fand the fused visible light feature F vi,f ;

[0053] Next, the fused infrared feature F ir,f With the fused visible light feature F vi,f The corresponding source infrared images I are respectively input into the reconstruction module RM and reconstructed. ir,c Visible light image I vi,c ;

[0054] At the same time, the fused infrared feature F ir,f With the fused visible light feature F vi,f The input is fed into the Feature Fusion Convolutional Module (FFCM) to perform feature fusion and obtain a fused feature map.

[0055] Then, the fused feature map is input into the Feature Reconstruction and Refinement (FRRB) module to generate the final fused image I. f ;

[0056] Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and thinning module FRRB are optimized.

[0057] The total loss function used in the first stage of training is defined as:

[0058]

[0059] in, It is the fusion loss, including pixel loss. Structural similarity loss and gradient loss

[0060] Fusion loss Defined as:

[0061]

[0062] Pixel loss Defined as:

[0063]

[0064] Structural similarity loss Defined as:

[0065]

[0066] SSIM calculates the structural similarity between input images.

[0067] Gradient loss Defined as:

[0068]

[0069] in, For Sobel gradient operators;

[0070] Reconstruction losses To ensure the accuracy of single-modal features, it is defined as:

[0071]

[0072] Among them, I ir,c and I vi,c For the reconstructed source infrared image and source visible light image, β, γ, and η are the weight balancing coefficients of the loss function, which are used as hyperparameters to adjust the weights of each loss term;

[0073] Step 6.2, the second stage of training the multi-view embedding module and the progressive multi-scale alignment module, specifically includes:

[0074] First, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are frozen after the first stage of training is completed;

[0075] Next, the unregistered infrared image was compared with the visible light image. Input feature encoder to extract fused misaligned infrared features and the fused visible light feature F vi,f ;

[0076] Then, the extracted features are input into the multi-view embedding module to generate pseudo-deformation fields from three views respectively. Next, pseudo-deformation fields at various scales are progressively applied to the unaligned infrared features to perform progressive feature alignment. The final aligned infrared features are obtained through the multi-scale, multi-view progressive alignment module (MSPA).

[0077] At the same time, the finally aligned infrared features Input the reconstruction module RM to reconstruct and align the image.

[0078] Next, the finally aligned infrared features and the fused visible light feature F vi,f The inputs are fed into the Feature Fusion Convolutional Module (FFCM) and the Feature Reconstruction and Refinement Module (FRRB) to generate a high-quality fused image after alignment and fusion.

[0079] To ensure the accuracy of feature alignment, a reconstruction loss function is defined. Calculate and reconstruct aligned images With strictly registered infrared images I ir The L1 distance between them is given by the formula:

[0080]

[0081] Meanwhile, to prevent unnatural distortions from arising in the pseudo-deformation field, the pseudo-deformation field predicted at the last scale is... Introducing smoothness loss, defined as:

[0082]

[0083] In addition, the pseudo-deformation field Applied to input misaligned infrared images To obtain the maximum correction result And by maximizing the correction results With strictly registered infrared images I ir The matching loss is defined using locally normalized cross-correlation (NCC).

[0084]

[0085] Finally, the total loss function of the multi-scale, multi-view progressive alignment module MSPA is defined as:

[0086]

[0087] Here, α is the weight hyperparameter of the smoothness loss term.

[0088] Furthermore, Step 7 includes:

[0089] First, the unregistered infrared image to be processed is compared with the visible light image. Input into the trained network;

[0090] Next, the feature encoder is used to extract the features of the fused image. Furthermore, infrared features are aligned using a multi-view embedding module and a progressive multi-scale alignment module.

[0091] Then, the finally aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction;

[0092] Finally, output a high-quality fused image after alignment and blending.

[0093] The present invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, the system comprising: a module for performing the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment.

[0094] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment.

[0095] The beneficial effects of this invention are:

[0096] 1. Improve the feature alignment accuracy and fusion quality of unregistered infrared and visible light images: This invention introduces multi-view pseudo-deformation field modeling, combined with local correlation calculation and global feature interaction, which can effectively guide the feature alignment of unregistered image pairs; the progressive multi-scale alignment module achieves feature alignment at different scales with a progressive alignment strategy, which significantly improves the consistency of cross-modal features and the structural fidelity of the fused image, and realizes the acquisition of high-quality fused images without strict registration.

[0097] 2. Enhance the robustness and generalization ability of unregistered fusion tasks: By collaborative modeling of multi-view pseudo-deformation fields and multi-scale progressive optimization, this invention reduces the impact of single deformation field prediction errors on the overall fusion effect and improves the fault tolerance and stability of the feature alignment process. In addition, by introducing local normalized cross-correlation loss and deformation field smoothness constraints, the generalization performance of the model under complex unregistered conditions is further enhanced.

[0098] 3. Optimize the training efficiency and inference effect of the fusion process: The present invention adopts a two-stage training strategy. In the first stage, the feature encoding and fusion module is trained on the registered data. In the second stage, the multi-view embedding and alignment module is specifically optimized on the unregistered data, which improves the overall training stability and convergence speed, and realizes the image registration and fusion task in an end-to-end manner. Moreover, the inference stage process is simple and can efficiently complete feature extraction, alignment, fusion and image reconstruction, taking into account both performance and inference efficiency.

[0099] 4. Improved adaptability and reliability in practical applications: The method of this invention can adapt to infrared and visible light image inputs with different registration error levels, and stably output high-quality fused images in diverse scenarios, demonstrating good potential for practical applications. Whether in applications such as security monitoring, night vision assistance, or complex environment perception, this invention can provide a more reliable and detailed cross-modal fusion solution, enhancing the applicability and practicality of the system.

[0100] 5. This invention fully leverages the advantages of deep learning in feature modeling and cross-modal alignment. Focusing on the key challenges of fusion of infrared and visible light images under unregistered conditions, it proposes a novel framework for multi-view pseudo-deformation field modeling and progressive multi-scale feature alignment, realizing end-to-end feature-level registration and fusion tasks. It solves the problems of insufficient cross-modal alignment accuracy and degraded fusion quality, and has significant theoretical value and broad practical application prospects. Attached Figure Description

[0101] Figure 1 This is a flowchart of the method of the present invention and a comparison diagram with existing inventions;

[0102] Figure 2 This is a structural diagram of the progressive multi-scale alignment module of the method of the present invention;

[0103] Figure 3 This is a diagram of the UIB Block structure, which is the main CNN branch of the method of this invention.

[0104] Figure 4 This is a structural diagram of the main Transformer branch Restormer of the method of the present invention;

[0105] Figure 5 This is a comparison chart of the performance of the method of the present invention with different registration methods;

[0106] Figure 6 This is a comparison chart of the performance of the method of the present invention with different fusion methods;

[0107] Figure 7 This is an overall flowchart of the framework of the unregistered image fusion method with multi-view embedding and progressive multi-scale alignment of the present invention. Detailed Implementation

[0108] Example 1: As Figures 1-7 As shown, an unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment is described, the method comprising:

[0109] Step 1: Obtain infrared and visible light image pairs, and preprocess the input images to unify image size and normalize them;

[0110] Furthermore, Step 1 includes:

[0111] The preprocessing includes two stages. In the first stage, the infrared and visible light images are randomly cropped to a size of 128×128 and normalized before being input into the feature encoder, feature fusion module, and multi-view embedding module. In the second stage, the infrared and visible light images are scaled to a size of 400×304, and an affine transformation is applied to the infrared image, including different degrees of rotation and translation. Specifically, the rotation angle is randomly sampled within the range of -2° to 2°, and the maximum translation amplitude in the horizontal and vertical directions is 2% of the image width and height, respectively. Then, an elastic transformation with a Gaussian function consisting of a kernel size of 63×63 and a standard deviation of 32 is applied to increase the diversity of training samples while simulating the offset situation in real scenes. After converting the RGB visible light images to the YCbCr color space in the preprocessing stage, the separated Y channel is fused with the infrared image, and then fused with the Cb and Cr channels and converted to the RGB color space to obtain the final fused image.

[0112] Step 2: Construct a feature encoder, which includes a convolutional neural network branch and a Transformer branch. The feature encoder extracts local detail features and global context features from infrared and visible light images, respectively. The extracted features are then concatenated and fused to establish a feature candidate library.

[0113] Furthermore, Step 2 includes:

[0114] Step 2.1: For the CNN branch extracting local image features and the Restormer branch extracting global image features, the input infrared image and visible light image are respectively input into the two branches of the feature encoder E for processing to obtain the preliminary extracted features, specifically including:

[0115] First, input the registered infrared image and the visible light image pair (I ir ,I vi ) and unregistered infrared images versus visible light images The input is fed into the Shallow Feature Extractor (SFE) to extract initial shallow features. Specifically, the SFE consists of four consecutive convolutional layers.

[0116] Next, the shallow features are input into the CNN branch and the Restormer branch respectively for deep feature extraction; the CNN branch uses the UIB Block designed based on MobileNetv4, and the Restormer branch uses the Restormer module for feature extraction.

[0117] Then, multi-scale feature representations of infrared and visible light images are obtained from the CNN branch and the Restormer branch, respectively.

[0118] Finally, the obtained features are denoted as follows: and

[0119] Among them, I ir Indicates a strictly registered infrared image, I vi Represents a visible light image. This represents an misaligned infrared image, E represents the feature encoder, SFE represents the shallow feature extractor, the CNN branch uses the UIB Block structure, and the Restormer branch uses the Restormer module. and These represent the features extracted from different inputs through the CNN branch and the Restormer branch, respectively.

[0120] Step 2.2: For feature fusion operations within the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representations; specifically including:

[0121] First, the features extracted from the infrared and visible light images in their respective modalities are concatenated.

[0122] Next, the concatenated features are further fused and adjusted using two convolutional blocks (CBs).

[0123] Then, the aligned and unaligned infrared and visible light features after fusion are denoted as F, respectively. ir,f , and F vi,f This is used for subsequent generalized deformation field prediction and fusion tasks;

[0124] Where CB represents the convolution block operation, F ir,f This indicates the fused infrared signature. F represents the fused unaligned infrared features. vi,f This indicates the visible light characteristics after fusion;

[0125] Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the effectiveness of the features; this includes:

[0126] The fused infrared feature F ir,f and the fused visible light feature F vi,f The data is input into the Reconstruction Module (RM) to reconstruct the corresponding source infrared image I. ir,c Visible light image I vi,cThe reconstruction module RM consists of six consecutive convolutional layers, with the last convolutional layer applying the Sigmoid activation function to transform the image into a normalized space.

[0127] Step 3: Construct a multi-view embedding module. Input the infrared and visible light features from the feature candidate library into the multi-view embedding module. Through correlation calculation and feature interaction, predict the pseudo-deformation fields of the first, second and third views respectively.

[0128] Furthermore, in Step 3, the operation process of the multi-view embedding module includes:

[0129] Unaligned infrared features fused from the feature candidate library With the fused visible light feature F vi,f The input is fed into a multi-scale feature extraction network to extract features from the final scale layer. and

[0130] First, the features of the final scale layer and The correlation within the local neighborhood is calculated to obtain the local correlation matrix R. Δi,Δj (i,j), local correlation matrix R Δi,Δj (i,j) is defined as:

[0131]

[0132] Where Δi,Δj∈{-1,0,1} represents the pixel displacement amplitude within the local range; i and j represent the i-th row and j-th column of the feature map, respectively; C is the total number of channels of the feature, and k is the channel index, corresponding to the k-th channel among the C channels;

[0133] Next, the local correlation matrix R of all locations is... Δi,Δj (i,j) are concatenated to form a complete correlation matrix R, defined as:

[0134] R = [R -1,-1 ,R -1,0 ,R -1,1 ,...,R 1,1 ]

[0135] Then, the complete correlation matrix R is compared with the features of the final scale layer. Feature concatenation is performed, and the concatenated features are fed into a 3×3 convolutional layer for feature fusion, followed by further feature extraction via the Restormer module. The extracted features are then input into three consecutive branches of a Generalized Deformation Field (GDF) network: the first branch directly feeds the extracted features into the GDF to predict the pseudo-deformation field from the first-viewpoint. The extracted features are then fed into the second branch, the UIB_Block module, where they are combined with channel attention to further extract and enhance features. Subsequently, a second-view pseudo-deformation field was obtained through another GDF prediction. The third branch stitches together the pseudo-deformation fields from the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo-deformation field from the third perspective.

[0136] In the above operation, the GDF network consists of a 3×3 convolutional block with custom weights. It predicts the pseudo-deformation field from three different perspectives at the Nth scale. The enhanced features obtained at the Nth scale will be passed to the next scale to enhance information transmission and feature fusion.

[0137] Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction in order from high to low levels, and gradually optimize the feature alignment effect at each scale.

[0138] Furthermore, in Step 4, the operation process of the progressive multi-scale alignment module includes:

[0139] First, the unaligned infrared features from the (N-1)th layer output by the multi-scale feature extraction network are... The pseudo-deformation field predicted using the previous scale For the registration of this misaligned feature, feature correction is performed sequentially from the highest layer to the lowest layer according to the scale number n = (N, N-1, ..., 1);

[0140] Next, at each scale, the pseudo-deformation field calculated based on the previous scale is... Unaligned infrared features of layer N-1 Perform progressive sampling transformation to obtain aligned infrared features.

[0141] Then, the aligned infrared features Input into the Restormer module at the current scale, and the visible light features at the same scale. Feature fusion is performed to improve the consistency of aligned features; the next step is to perform alignment of infrared features. Enhanced features compared to the previous scale The data is then fed into the second Restormer module for feature fusion to further enhance the information transfer between features at different scales. Step 3 is repeated to obtain the pseudo-deformation fields from three different perspectives at the corresponding scales.

[0142] Finally, feature correction and fusion at all scales are performed sequentially to progressively optimize the alignment of infrared and visible light features across multiple scales, providing a precise feature foundation for subsequent fusion tasks. This progressive multi-scale feature alignment process yields the final aligned infrared features. The process is defined as:

[0143]

[0144] Where n represents the scale number and N represents the total number of scale layers. This represents the unaligned infrared feature of the nth layer. This represents the pseudo-deformation field predicted at the nth layer, where W is the Warp operation, which involves sampling and transforming the pseudo-deformation field. This represents the visible light characteristics of the nth layer. This represents the aligned infrared features after applying a pseudo-deformation field transformation. It is the final aligned infrared feature.

[0145] Step 5: Construct a feature fusion module. Input the progressively aligned infrared and visible light features into the feature fusion convolution module. Perform feature fusion through convolution operations to generate a fused feature map. Input the fused feature map into the feature reconstruction and thinning module. Use convolution and activation operations to generate the final fused image.

[0146] Furthermore, in Step 5, the operation process of the feature fusion module includes:

[0147] First, the fused infrared features F that are aligned with each other are... ir,f With the fused visible light feature F vi,f The input is fed into the Feature Fusion Convolutional Module (FFCM), where features are fused through convolution operations to generate a fused feature map.

[0148] Next, the fused feature map is input into the Feature Reconstruction and Refinement Block (FRRB). In the FRRB module, the feature representation capability is first enhanced by the Channel Attention (CA) module. Then, features are further extracted by a convolutional block containing a 3×3 convolutional layer, a Batch Normalization (BN) layer, and a LeakyReLU (LReLU) activation function. Finally, a 1×1 convolutional layer followed by a Sigmoid activation function is used to output the final fused image I. f The feature fusion and reconstruction process is defined as follows:

[0149] I f =FRRB(FFCM(F ir,f ,F vi,f )),

[0150] Wherein, FFCM represents the feature fusion convolution module, FRRB represents the feature reconstruction and refinement module, and I f This represents the final merged image.

[0151] Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module.

[0152] Furthermore, Step 6 includes:

[0153] Step 6.1: The first stage involves training the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including:

[0154] First, the registered infrared image is paired with the visible light image (I ir ,I vi The fused infrared features F are input into the feature encoder E and extracted respectively. ir,f and the fused visible light feature F vi,f ;

[0155] Next, the fused infrared feature F ir,f With the fused visible light feature F vi,f The corresponding source infrared images I are respectively input into the ReconstructionModule (RM) and reconstructed. ir,c Visible light image I vi,c ;

[0156] At the same time, the fused infrared feature F ir,f With the fused visible light feature Fvi,f The input is fed into the Feature Fusion Convolutional Module (FFCM) to perform feature fusion and obtain a fused feature map.

[0157] Then, the fused feature map is input into the Feature Reconstruction and Refinement (FRRB) module to generate the final fused image I. f ;

[0158] Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and thinning module FRRB are optimized.

[0159] The total loss function used in the first stage of training is defined as:

[0160]

[0161] in, It is the fusion loss, including pixel loss. Structural similarity loss and gradient loss

[0162] Fusion loss Defined as:

[0163]

[0164] Pixel loss Defined as:

[0165]

[0166] Structural similarity loss Defined as:

[0167]

[0168] SSIM calculates the structural similarity between input images.

[0169] Gradient loss Defined as:

[0170]

[0171] in, For Sobel gradient operators;

[0172] Reconstruction losses To ensure the accuracy of single-modal features, it is defined as:

[0173]

[0174] Among them, I ir,c and Ivi,c For the reconstructed source infrared image and source visible light image, β, γ, and η are the weight balancing coefficients of the loss function, which are used as hyperparameters to adjust the weights of each loss term;

[0175] Step 6.2, the second stage of training the multi-view embedding module and the progressive multi-scale alignment module, specifically includes:

[0176] First, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are frozen after the first stage of training; only the multi-scale multi-view progressive alignment module (MSPA) is trained.

[0177] Next, the unregistered infrared image was compared with the visible light image. Input feature encoder to extract fused misaligned infrared features and the fused visible light feature F vi,f ;

[0178] Then, the extracted features are input into the multi-view embedding module to generate pseudo-deformation fields from three views respectively. Next, feature transfer is performed step by step, applying pseudo-deformation fields at various scales to the unaligned infrared features for progressive feature alignment. The final aligned infrared features are obtained through the multi-scale, multi-view progressive alignment module MSPA.

[0179] At the same time, the finally aligned infrared features Input the reconstruction module RM to reconstruct the aligned image.

[0180] Next, the finally aligned infrared features and the fused visible light feature F vi,f The inputs are fed into the Feature Fusion Convolutional Module (FFCM) and the Feature Reconstruction and Refinement Module (FRRB) to generate a high-quality fused image after alignment and fusion.

[0181] To ensure the accuracy of feature alignment, a reconstruction loss function is defined. Calculate and reconstruct aligned images With strictly registered infrared images I ir The L1 distance between them is given by the formula:

[0182]

[0183] Meanwhile, to prevent unnatural distortions from arising in the pseudo-deformation field, the pseudo-deformation field predicted at the last scale is... Introducing smoothness loss, defined as:

[0184]

[0185] In addition, the pseudo-deformation field Applied to input misaligned infrared images To obtain the maximum correction result And by maximizing the correction results With strictly registered infrared images I ir The matching loss is defined using locally normalized cross-correlation (NCC).

[0186]

[0187] Finally, the total loss function of the multi-scale, multi-view progressive alignment module MSPA is defined as:

[0188]

[0189] Here, α is the weight hyperparameter of the smoothness loss term.

[0190] Step 7: Input the unregistered infrared and visible light images to be processed into the trained network, and output the high-quality fused image after alignment and fusion.

[0191] Furthermore, Step 7 includes:

[0192] First, the unregistered infrared image to be processed is compared with the visible light image (I1). ir ,I vi The input is fed into the trained network;

[0193] Next, the feature encoder is used to extract the features of the fused image. Furthermore, infrared features are aligned using a multi-view embedding module and a progressive multi-scale alignment module.

[0194] Then, the finally aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction;

[0195] Finally, output a high-quality fused image after alignment and blending.

[0196] Meanwhile, without activating the Multi-Scale Multi-View Progressive Alignment Module (MSPA), the model can input aligned infrared and visible light image pairs (I... ir ,I vi ), to obtain a high-quality fused image I in the aligned scene. f .

[0197] This invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized in that the system comprises:

[0198] The image acquisition and preprocessing module is used to acquire infrared and visible light image pairs, and to preprocess the input images, including unifying image size and normalizing the images.

[0199] The feature candidate library building module is used to construct a feature encoder, which includes a convolutional neural network branch and a Transformer branch. These branches extract local detail features and global context features from infrared and visible light images, respectively, and then concatenate and fuse the extracted features to build a feature candidate library.

[0200] The multi-view embedding module is used to input infrared and visible light features from the feature candidate library into the multi-view embedding module. Through correlation calculation and feature interaction, it predicts the pseudo-deformation fields of the first, second and third views respectively.

[0201] The progressive multi-scale alignment module is used to apply pseudo-deformation fields at various scales to infrared features, perform feature correction in order from high to low levels, and gradually optimize the feature alignment effect at each scale.

[0202] The feature fusion module is used to input progressively aligned infrared and visible light features into the feature fusion convolution module, perform feature fusion through convolution operations, and generate a fused feature map; the fused feature map is then input into the feature reconstruction and thinning module, which uses convolution and activation operations to generate the final fused image;

[0203] The training module is used to train the feature encoder and feature fusion module using registered image pairs in the first stage, and the multi-view embedding module and progressive multi-scale alignment module using unregistered image pairs in the second stage.

[0204] The generation module is used to input the unregistered infrared and visible light images to be processed into the trained network and output a high-quality fused image after alignment and fusion.

[0205] Furthermore, the present invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized in that the system comprises:

[0206] The preprocessing module preprocesses the infrared and visible light image pairs by performing channel separation, random cropping, scaling, applying affine and elastic transformations, and normalization operations to meet the model input requirements.

[0207] The shallow feature extraction network performs preliminary feature extraction on the input infrared and visible light image pairs, transforms the low-dimensional image into high-dimensional features, mines shallow features in the image, and provides basic feature requirements for subsequent feature extraction networks.

[0208] The backbone feature extraction network consists of a sub-network composed of CNNs to extract local features and a sub-network composed of Restormers to extract global features. It is used to feed shallow features into the corresponding sub-networks to further mine global and local information in image features.

[0209] A multi-scale, multi-view progressive alignment network is used to extract features at different scales of an image. Then, at different scales, a CNN sub-network and a Restormer self-network are used to predict pseudo-deformation fields at multiple viewpoints. Next, the pseudo-deformation fields are applied to the offset infrared image features to obtain the registered and aligned infrared image features. The features are predicted layer by layer to obtain the final scale multi-view pseudo-deformation field, which is applied to the original offset infrared image features to obtain the final scale-aligned infrared image features.

[0210] The reconstruction network can reconstruct the input features from high dimensions into the corresponding modal image through the convolution module, and can encourage the network to retain the original image semantic information as much as possible.

[0211] The feature fusion network can perform cross-modal fusion of offset infrared image features and visible light image features, fully explore feature information in different modalities, and effectively fuse features from two modalities, thereby improving the information flow between features.

[0212] The fused image reconstruction network reconstructs images based on fusion features, and can obtain a final fused image based on the fusion features. The fused image contains detailed information from the visible light image and salient targets from the infrared image.

[0213] The training module, through different loss function designs, can train sub-networks in a two-stage manner, including: shallow feature extraction network, backbone feature extraction network, multi-scale multi-view progressive alignment network, reconstruction network, feature fusion network, and fused image reconstruction network.

[0214] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment.

[0215] This invention innovatively introduces a multi-view, multi-scale feature-level pseudo-deformation field prediction method. First, fused features for corresponding modes are obtained based on global and local feature extraction. Next, the features are interacted with the global context through local correlation to generate pseudo-deformation fields from different perspectives, collaboratively guiding feature-level alignment. Simultaneously, a multi-scale progressive alignment strategy is adopted, gradually refining and correcting offset features at different scales, significantly reducing dependence on a single deformation field and accuracy requirements. A two-stage training paradigm further enhances the effectiveness of feature-level registration. Based on this, a feature fusion module and a feature reconstruction module achieve full integration and fine-grained restoration of infrared and visible light information, ultimately outputting a high-quality fused image with rich details and consistent structure.

[0216] To verify the effectiveness of the method of this invention, training data was constructed based on the RoadScene dataset. The RoadScene dataset contains 221 registered infrared and visible light image pairs, of which 200 pairs were randomly selected as the training set and the remaining 21 pairs as the test set. To further evaluate the robustness and generalization ability of the method of this invention, 50 pairs and 40 pairs of infrared-visible light image pairs were randomly selected from the M3FD dataset and the MSRS dataset, respectively, as additional test sets. During training and testing, to simulate registration errors that may occur in real-world applications, this invention generates unregistered samples through affine transformation and elastic deformation. The affine transformation includes random rotation and translation, with rotation angles ranging from -2° to 2°, and horizontal and vertical translation amplitudes up to 2% of the image width and height. Elastic deformation uses a Gaussian function with a kernel size of 63×63 and a standard deviation of 32 to generate random local deformation, thereby simulating nonlinear local distortion phenomena. The method of this invention is developed based on the PyTorch framework and trained and tested in a hardware environment equipped with an NVIDIA GeForce RTX 4090 graphics card. The model training consisted of two phases: Phase 1 used registered image pairs as input to train the Feature Encoder (FE) and Feature Decoder (FD). In this phase, the input images were first randomly cropped to 128×128 pixels, with a batch size of 10 and an initial learning rate of 1e-2. Cosine annealing was used to gradually decrease the learning rate, and the total training duration was 2000 epochs. Phase 2 used unregistered image pairs as input to optimize only the parameters of the Progressive Multi-Scale Alignment (MSPA) module, freezing the parameters of the other modules. In this phase, the input image size was uniformly adjusted to 400×304 pixels, with a batch size of 4 and an initial learning rate of 1e-4. Cosine annealing was continued for further adjustments, and the total training duration was 4000 epochs, with each epoch containing 50 iterations. During training, different affine and elastic transformations were continuously applied to effectively expand the number of training samples, ultimately exceeding 800,000 samples. The training strategy of this invention not only effectively improves the feature alignment and fusion performance of the model under unregistered conditions, but also significantly enhances the robustness and generalization ability of the model under different scenarios and degradation conditions.

[0217] To further explain, this invention employs mutual information (MI), cross-entropy (CE), visual information fidelity (VIF), and gradient-based fusion evaluation metrics (Q) when evaluating image fusion quality. AB / F ), evaluation metrics based on edge features (Q) CB ) and texture feature-based evaluation metrics (Q CVThese six key metrics are used to evaluate the quality of the fused image. Mutual information (MI) measures the amount of information from the source image contained in the fused image; a higher MI value indicates that the fused image retains more effective information from the source image. Cross-entropy (CE) primarily reflects the consistency of information distribution between the fused and source images; a lower CE value indicates a better fusion effect. Visual information fidelity (VIF) evaluates the degree to which the fused image retains the features of the source image at the visual perception level; a higher VIF value indicates that the fused image is visually closer to the source image. Gradient-based fusion evaluation metrics (Q...) AB / F Focusing on the preservation of gradient information in the fused image can effectively reflect the degree of retention of image edge details and sharpness. Q AB / F A higher value indicates better preservation of edge structure in the fused image; the evaluation metric based on edge features (Q) CB Q is used to measure the correlation between edge features in the fused image and the source image. CB A higher value indicates that the fused image better preserves the edge details of the source image; the evaluation metric based on texture features (Q) CV This reflects the completeness and richness of texture features in the fused image, Q. CV A higher value indicates better performance in terms of texture detail in the fused image. Through comprehensive evaluation of the above six indicators, this invention can comprehensively and objectively assess the performance of the fused image in terms of information preservation, structural consistency, and visual perception quality, ensuring that the fusion result not only has high-quality detail preservation but also meets the perceptual requirements of the human visual system.

[0218] Table 1. Quantitative comparative experimental results of registration and fusion methods on multiple datasets.

[0219]

[0220]

[0221] Furthermore, the image fusion method proposed in this invention was compared with existing advanced methods such as UMF-CMGR, SuperFusion, MURF, SemLA, RFIVF, IMF, and IVFWSR on the RoadScene, M3FD, and MSRS datasets. The comparison results are shown in Table 1. The method of this invention achieved excellent performance in all metrics on the above three datasets, especially in terms of information preservation, structure preservation, and visual quality, which are significantly better than the comparison methods.

[0222] To further verify the effectiveness of this method in misaligned scenarios, the registered and aligned infrared features were... The reconstructed aligned image is obtained after the feature reconstruction module RM. Then, with strictly registered infrared images I ir Several metrics were calculated to evaluate the image alignment performance. These metrics include: MI (Mean Integrity), which measures the consistency of shared content between two images based on the commonality of information; MS-SSIM (Multi-Scale Similarity), which assesses perceptual similarity based on multi-scale structure; and NCC (Normalized Cross-Correlation Coefficient), which characterizes the linear correlation of image intensity. Higher values ​​for these metrics generally indicate better image registration performance. Specific experimental results are shown below. Figure 5 As shown.

[0223] Qualitative experimental results compared with the above methods are as follows: Figure 6 As shown, the experimental results of this invention achieve higher quality fusion results and effectively alleviate image offset and visual misalignment. The above experimental results fully verify the effectiveness of the method of this invention under different datasets and scene conditions, demonstrating good fusion performance and excellent cross-dataset generalization ability, further reflecting the application potential and practical value of this invention in the task of fusing unregistered infrared and visible light images.

[0224] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment, characterized in that: The method includes: Step 1: Obtain infrared and visible light image pairs, and preprocess the input images to unify image size and normalize them; Step 2: Construct a feature encoder, which includes a convolutional neural network branch and a Restormer branch. The feature encoder extracts local detail features and global context features from infrared and visible light images, respectively. The extracted features are then concatenated and fused to establish a feature candidate library. Step 3: Construct a multi-view embedding module. Input the infrared and visible light features from the feature candidate library into the multi-view embedding module. Through correlation calculation and feature interaction, predict the pseudo-deformation fields of the first, second and third views at each scale. Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction in order from high to low levels, and gradually optimize the feature alignment effect at each scale. Step 5: Construct a feature fusion module. Input the progressively aligned infrared and visible light features into the feature fusion convolution module. Perform feature fusion through convolution operations to generate a fused feature map. Input the fused feature map into the feature reconstruction and thinning module. Use convolution and activation operations to generate the final fused image. Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module. Step 7: Input the unregistered infrared and visible light images to be processed into the trained network, and output the high-quality fused image after alignment and fusion.

2. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 1 includes: The preprocessing includes two stages. In the first stage, the infrared and visible light images are randomly cropped to a size of 128×128 and normalized before being input into the feature encoder, feature fusion module, and multi-view embedding module. In the second stage, the infrared and visible light images are scaled to a size of 400×304, and an affine transformation is applied to the infrared images, including different degrees of rotation and translation. Then, an elastic transformation is applied with a Gaussian function consisting of a kernel size of 63×63 and a standard deviation of 32 to increase the diversity of training samples while simulating the offset in real-world scenarios.

3. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 2 includes: Step 2.1: For feature extraction operations in the CNN and Restormer branches, the input infrared and visible light images are respectively input into the two branches of the feature encoder E for processing to obtain preliminary extracted features, specifically including: First, the input registered infrared image and the visible light image are compared. And unregistered infrared images versus visible light images The input is fed into the Shallow Feature Extractor (SFE) to extract initial shallow features; Next, the shallow features are input into the CNN branch and the Restormer branch respectively for deep feature extraction; Then, multi-scale feature representations of infrared and visible light images are obtained from the CNN branch and the Restormer branch, respectively. Finally, the obtained features are denoted as follows: , , and ; in, Indicates a strictly registered infrared image. Represents a visible light image. This represents an misaligned infrared image, E represents the feature encoder, SFE represents the shallow feature extractor, the CNN branch uses the UIB Block structure, and the Restormer branch uses the Restormer module. and These represent the features extracted from different inputs through the CNN branch and the Restormer branch, respectively. Step 2.2: For feature fusion operations within the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representations; specifically including: First, the features extracted from the infrared image and the visible light image in their respective modalities are stitched together. Next, the concatenated features are further fused and adjusted using two convolutional blocks (CBs). Then, the aligned and unaligned infrared and visible light features after fusion are denoted as follows: , and This is used for subsequent generalized deformation field prediction and fusion tasks; Where CB represents the convolution block operation. This indicates the fused infrared signature. This indicates the unaligned infrared features after fusion. This indicates the visible light characteristics after fusion; Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the effectiveness of the features; this includes: The fused infrared features and the fused visible light characteristics The images are input into the reconstruction module RM to reconstruct the corresponding source infrared images. Visible light image source .

4. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 3, the operation process of the multi-view embedding module includes: Unaligned infrared features fused from the feature candidate library With the fused visible light characteristics The input is fed into a multi-scale feature extraction network to extract features from the final scale layer. and ; First, the features of the final scale layer and The correlation within the local neighborhood is calculated to obtain the local correlation matrix. Local correlation matrix Defined as: ； in Represents the pixel displacement amplitude within a local range; i and j represent the i-th row and j-th column of the feature map, respectively; C is the total number of channels in the feature map, and k is the channel index, corresponding to the k-th channel out of the C channels; Next, the local correlation matrix of all locations is... The data is then combined to form a complete correlation matrix. Defined as: ； Then, the complete correlation matrix Features of the final scale layer , Feature concatenation is performed, and the concatenated features are fed into a 3×3 convolutional layer and a Restormer module for further feature extraction. Then, the extracted features are input into three consecutive branches of the pseudo-deformation field prediction network (GDF): the first branch directly feeds the extracted features into the GDF for processing, predicting the pseudo-deformation field from the first-viewpoint. The extracted features are then fed into the second branch, the UIB_Block module, where they are further enhanced by combining channel attention to extract more features. Subsequently, a second-view pseudo-deformation field was obtained through another GDF prediction. The third branch stitches together the pseudo-deformation fields from the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo-deformation field from the third perspective. ; In the above operation, the GDF network consists of a 3×3 convolutional block with custom weights. It predicts the pseudo-deformation field from three different perspectives at the Nth scale. The enhanced features obtained at the Nth scale will be passed to the next scale to enhance information transmission and feature fusion.

5. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 4, the operation process of the progressive multi-scale alignment module includes: First, the unaligned infrared features of the (n-1)th layer output by the multi-scale feature extraction network are... The pseudo-deformation field predicted using the previous scale Registration for this misaligned feature is performed according to scale numbering. The feature correction is performed sequentially from the highest layer to the lowest layer. Next, at each scale, the pseudo-deformation field calculated based on the previous scale is... Unaligned infrared features of layer n-1 Perform progressive sampling transformation to obtain aligned infrared features. ; Then, the aligned infrared features Input into the Restormer module at the current scale, and the visible light features at the same scale. Feature fusion is performed to improve the consistency of aligned features; the next step is to perform alignment of infrared features. Enhanced features compared to the previous scale The feature is then fed into the second Restormer module for feature fusion to further enhance the information transfer between features at different scales. Step 3 is repeated to obtain the pseudo-deformation fields from three different perspectives at the corresponding scale. Finally, feature correction and fusion at all scales are performed sequentially to progressively optimize the alignment of infrared and visible light features across multiple scales, providing a precise feature foundation for subsequent fusion tasks. This progressive multi-scale feature alignment process yields the final aligned infrared features. The process is defined as: ； in, Indicates the scale number. Indicates the total number of scale layers. Indicates the first The misaligned infrared features of the layer Indicates the first The pseudo-deformation field predicted by the layer, It's a Warp operation, where Warp represents a sampling transformation based on the pseudo-deformation field. Indicates the first Visible light characteristics of the layer, This represents the aligned infrared features after applying a pseudo-deformation field transformation. It is the final aligned infrared feature.

6. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 5, the operation process of the feature fusion module includes: First, the fused infrared features that are aligned with each other are... With the fused visible light characteristics The input is fed into the Feature Fusion Convolutional Module (FFCM), where features are fused through convolution operations to generate a fused feature map. Next, the fused feature map is input into the Feature Reconstruction and Refinement (FRRB) module. In the FRRB module, the feature representation capability is first enhanced by the Channel Attention (CA) module. Then, features are further extracted by a convolutional block containing a 3×3 convolutional layer, a batch normalization (BN) layer, and a Leaky ReLU activation function. Finally, a 1×1 convolutional layer followed by a Sigmoid activation function is used to output the final fused image. The feature fusion and reconstruction process is defined as follows: ； In this context, FFCM represents the feature fusion convolution module, and FRRB represents the feature reconstruction and refinement module. This represents the final merged image.

7. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 6 includes: Step 6.1: The first stage involves training the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including: First, the registered infrared image is compared with the visible light image. The fused infrared features are input into the feature encoder E and extracted separately. and the fused visible light characteristics ; Next, the fused infrared features With the fused visible light characteristics The corresponding source infrared images are reconstructed by inputting them into the reconstruction module RM. Visible light image source ; At the same time, the fused infrared features With the fused visible light characteristics The input is fed into the Feature Fusion Convolutional Module (FFCM) to perform feature fusion and obtain a fused feature map. Then, the fused feature map is input into the Feature Reconstruction and Thinning (FRRB) module to generate the final fused image. ; Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and thinning module FRRB are optimized. The total loss function used in the first stage of training is defined as: ； in, It is the fusion loss, including pixel loss. Structural similarity loss and gradient loss ; Fusion loss Defined as: ； Pixel loss Defined as: ； Structural similarity loss Defined as: ； SSIM calculates the structural similarity between input images; Gradient loss Defined as: ； in, For Sobel gradient operators; Reconstruction losses To ensure the accuracy of single-modal features, it is defined as: ； in, and For the reconstructed source infrared image and source visible light image, , , This is the weight balancing coefficient of the loss function, used as a hyperparameter to adjust the weights of each loss term; Step 6.2, the second stage of training the multi-view embedding module and the progressive multi-scale alignment module, specifically includes: First, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are frozen after the first stage of training is completed; Next, the unregistered infrared image was compared with the visible light image. Input feature encoder to extract fused misaligned infrared features and the fused visible light characteristics ; Then, the extracted features are input into the multi-view embedding module to generate pseudo-deformation fields from three views respectively. Next, pseudo-deformation fields at various scales are progressively applied to the unaligned infrared features to perform progressive feature alignment. The final aligned infrared features are obtained through the multi-scale, multi-view progressive alignment module (MSPA). ; At the same time, the finally aligned infrared features Input the reconstruction module RM to reconstruct the aligned image. ; Next, the finally aligned infrared features and the fused visible light characteristics The inputs are fed into the Feature Fusion Convolutional Module (FFCM) and the Feature Reconstruction and Refinement Module (FRRB) to generate a high-quality fused image after alignment and fusion. ; To ensure the accuracy of feature alignment, a reconstruction loss function is defined. Calculate and reconstruct aligned images With strictly registered infrared images The L1 distance between them is given by the formula: ； Meanwhile, to prevent unnatural distortions from arising in the pseudo-deformation field, the pseudo-deformation field predicted at the last scale is... Introducing smoothness loss, defined as: ； In addition, the pseudo-deformation field Applied to input misaligned infrared images To obtain the maximum correction result And by maximizing the correction results With strictly registered infrared images The matching loss is defined using locally normalized cross-correlation (NCC). ； Finally, the total loss function of the multi-scale, multi-view progressive alignment module MSPA is defined as: ； in, It is the weight hyperparameter of the smoothness loss term.

8. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 7 includes: First, the unregistered infrared image to be processed is compared with the visible light image. Input into the trained network; Next, the feature encoder is used to extract the features of the fused image. Furthermore, infrared features are aligned using a multi-view embedding module and a progressive multi-scale alignment module. Then, the finally aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction; Finally, output a high-quality fused image after alignment and blending. .

9. An unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized in that, The system includes a module for performing the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment as described in any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment as described in any one of claims 1 to 8.

Images