Unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment

Abstract

The invention relates to an unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment, and belongs to the technical field of computer vision and image processing. The method comprises the following steps: constructing a feature encoder, extracting multi-scale features of an infrared image and a visible light image, and establishing a feature candidate library; inputting the unregistered infrared features and visible light features into a multi-view embedding module, and predicting a plurality of pseudo-deformation fields; the pseudo-deformation field is applied to the infrared features, and cross-scale progressive alignment is carried out; inputting the aligned infrared features and visible light features into a feature fusion module, and extracting fusion features; inputting the fusion features into a reconstruction module to generate a final fusion image; training a feature encoder, a feature fusion convolution module and an image reconstruction and refinement module; and the trained network is used for fusion processing of infrared and visible light images. According to the method, the fusion quality of the infrared and visible light images is improved in an unaligned scene, and the robustness of feature alignment and the definition of image details are enhanced.

Description

Technical Field

[0001] The present invention relates to an unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment, and belongs to the technical field of computer vision and image processing. Background Art

[0002] With the widespread application of infrared and visible light imaging technologies, the fusion of infrared and visible light images has become a key approach to improving image perception quality and environmental adaptability. By fusing the thermal radiation information in infrared images with the rich texture details in visible light images, more complete, clear, and robust visual information can be obtained in various complex environments. However, in practical applications, due to various factors such as imaging device differences, viewing angle offset, and optical distortion, infrared and visible light images often suffer from registration errors, resulting in poor or even ineffective performance of traditional registration-based fusion methods. Existing fusion methods typically rely on strict registration assumptions, making it difficult to achieve high-precision fusion in the presence of local displacement and global pixel elastic deformation. In particular, direct image fusion methods under unregistered conditions are prone to introducing artifacts, blurring, and pixel structure misalignment, which seriously affect the visual quality of the fused image and the accuracy of subsequent tasks such as detection and recognition.

[0003] Specifically, the present invention proposes a method for fusion of unregistered infrared and visible light images based on multi-view embedding and progressive multi-scale alignment, and innovatively introduces a multi-view-multi-scale feature-level pseudo-deformation field prediction method. Summary of the Invention

[0004] The technical problem solved by the present invention is: the present invention provides an unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment, which is used to overcome the problems of insufficient alignment accuracy and reduced fusion quality in the fusion of unregistered infrared and visible light images. The method of the present invention effectively improves the consistency of cross-modal features and the quality of the fused image through multi-view pseudo-deformation field prediction and multi-scale progressive feature alignment strategy, and can achieve high-precision fusion without strict alignment.

[0005] The technical solution of the present invention is: a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment, the method comprising:

[0006] Step 1: Obtain infrared image and visible light image pairs, and preprocess the input images to unify the image size and normalize them;

[0007] Step 2: Construct a feature encoder. The feature encoder includes a convolutional neural network branch and a Transformer branch. It extracts local detail features and global context features of infrared images and visible light images respectively, and then concatenates and fuses the extracted features to establish a feature candidate library.

[0008] Step 3: Construct a multi-view embedding module, input the infrared features and visible light features in the feature candidate library into the multi-view embedding module, and predict the pseudo-deformation fields of the first, second, and third perspectives respectively through correlation calculation and feature interaction;

[0009] Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction from high-level to low-level, and gradually optimize the feature alignment effect at each scale;

[0010] Step 5: Construct a feature fusion module, input the progressively aligned infrared features and visible light features into the feature fusion convolution module, perform feature fusion through convolution operation, and generate a fused feature map; input the fused feature map into the feature reconstruction and refinement module, and generate the final fused image through convolution and activation operations;

[0011] Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module.

[0012] Step 7: Input the unregistered infrared and visible light images to be processed into the trained network and output the aligned and fused high-quality fused images.

[0013] Furthermore, the Step 1 includes:

[0014] The preprocessing consists of two stages. In the first stage, the infrared images and visible light images are randomly cropped to a size of 128×128, and the images are normalized before being input into the feature encoder, feature fusion module, and multi-view embedding module. In the second stage, the infrared images and visible light images are scaled to a size of 400×304, and an affine transformation is applied to the infrared images, including different degrees of rotation and translation. Then, an elastic transformation is applied with a Gaussian function with a kernel size of 63×63 and a standard deviation of 32 to increase the diversity of training samples while simulating the offset conditions in real scenes.

[0015] Furthermore, the Step 2 includes:

[0016] Step 2.1: For the feature extraction operation of the CNN branch and the Restormer branch, the input infrared image and visible light image are respectively input into the two branches of the feature encoder E for processing to obtain the preliminary extracted features, specifically including:

[0017] First, the input registered infrared image and visible light image pair (I ir ,I vi) and unregistered infrared and visible light images Input into the shallow feature extractor SFE to extract the initial shallow features;

[0018] Then, the shallow features are input into the CNN branch and Restormer branch respectively for deep feature extraction;

[0019] Then, multi-scale feature representations of infrared images and visible light images are obtained from the CNN branch and Restormer branch respectively;

[0020] Finally, the features obtained are recorded as and

[0021] Among them, I ir represents the strictly registered infrared image, I vi represents a visible light image, represents an unaligned infrared image, E represents a feature encoder, SFE represents a shallow feature extractor, the CNN branch adopts the UIB Block structure, and the Restormer branch adopts the Restormer module. and Represents the features extracted by the CNN branch and the Restormer branch for different inputs respectively;

[0022] Step 2.2: For the feature fusion operation under the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representation; specifically, the following steps are performed:

[0023] First, the features extracted from the infrared image and visible light image in their respective modalities are spliced separately;

[0024] Then, the spliced features are passed through two convolution blocks CB for further feature fusion and feature adjustment;

[0025] Then, the aligned and unaligned infrared features and visible light features after fusion are denoted as F ir,f 、 and F vi,f , used for subsequent generalized deformation field prediction and fusion tasks;

[0026] Among them, CB represents the convolution block operation, F ir,f represents the fused infrared features, represents the fused unaligned infrared features, F vi,f represents the fused visible light features;

[0027] Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the validity of the features. This includes:

[0028] The fused infrared feature F ir,f and the fused visible light feature F vi,f Input into the reconstruction module RM to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c .

[0029] Furthermore, in Step 3, the operation process of the multi-view embedding module includes:

[0030] The unaligned infrared features fused in the feature candidate library and the fused visible light feature F vi,f Input into the multi-scale feature extraction network to extract the features of the final scale layer and

[0031] First, the features of the final scale layer and Perform correlation calculation within the local neighborhood to obtain the local correlation matrix R Δi,Δj (i,j), local correlation matrix R Δi,Δj (i,j) is defined as:

[0032]

[0033] Where Δi, Δj∈{-1,0,1} represents the pixel displacement amplitude in the local range; i and j represent the i-th row and j-th column of the feature map respectively; C is the total number of channels of the feature, and k is the channel index, corresponding to the k-th channel in C;

[0034] Next, the local correlation matrix R of all positions Δi,Δj (i, j) are spliced to form a complete correlation matrix R, which is defined as:

[0035] R=[R -1,-1 ,R -1,0 ,R -1,1 ,...,R 1,1 ]

[0036] Then, the complete correlation matrix R is combined with the features of the final scale layer The features are spliced and sent to the 3×3 convolution layer and the Restormer module for further feature extraction. After that, the extracted features are input into the pseudo deformation field prediction network GDF of three consecutive different branches: the first branch directly sends the extracted features to the GDF for processing to predict the pseudo deformation field of the first perspective. The extracted features are then fed into the second branch UIB_Block module to further extract enhanced features in combination with channel attention. Then, another GDF is used to predict the second-perspective pseudo-deformation field. The third branch splices the pseudo deformation field of the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo deformation field of the third perspective.

[0037] In the above operation, the GDF network consists of a 3×3 convolution block with custom weights. It is to predict the pseudo deformation field of three different perspectives at the Nth scale, It is the enhanced feature obtained at the Nth scale, which will be passed to the next scale for enhanced information transmission and feature fusion.

[0038] Furthermore, in Step 4, the operation process of the progressive multi-scale alignment module includes:

[0039] First, the N-1th layer unaligned infrared features output by the multi-scale feature extraction network are The pseudo deformation field predicted by the previous scale For the registration of the misaligned features, feature correction is performed in sequence from the highest level to the lowest level in the order of scale numbers n=(N, N-1, ..., 1);

[0040] Then, at each scale, the pseudo deformation field calculated at the previous scale is Unaligned infrared features of layer N-1 Perform progressive sampling transformation to obtain aligned infrared features

[0041] Then, the aligned infrared features Input to the Restormer module of the current scale, and the visible light features at the same scale Perform feature fusion to improve the consistency of the aligned features; then the aligned infrared features Enhanced features from the previous scale Continue to send it to the second Restormer module for feature fusion to further enhance the information transmission of features between different scales, and repeat Step 3 to obtain the pseudo deformation fields of three different perspectives at the corresponding scales.

[0042] Finally, the feature correction and fusion at all scales are completed in sequence, and the alignment effect of infrared and visible light features in multi-scale space is gradually optimized, providing a feature basis for accurate matching for subsequent fusion tasks; the progressive multi-scale feature alignment process obtains the final aligned infrared feature. The process is defined as:

[0043]

[0044] Among them, n represents the scale number, N represents the total number of scale layers, represents the unaligned infrared features of layer n, Represents the pseudo deformation field predicted at the nth layer, W is the Warp operation, and Warp represents the operation of sampling transformation based on the pseudo deformation field. represents the visible light characteristics of the nth layer, represents the aligned infrared features after applying the pseudo deformation field transformation, is the final aligned infrared signature.

[0045] Furthermore, in Step 5, the operation process of the feature fusion module includes:

[0046] First, the aligned fused infrared features F ir,f and the fused visible light feature F vi,f Input to the feature fusion convolution module FFCM, perform feature fusion through convolution operation, and generate a fused feature map;

[0047] Next, the fused feature map is input into the feature reconstruction and refinement module FRRB. In the FRRB module, the feature representation capability is first enhanced by the channel attention module CA, and then the features are further extracted by a convolution block containing a 3×3 convolution layer, a batch normalization BN layer and a LeakyReLU activation function. Finally, a 1×1 convolution layer is used with a Sigmoid activation function to output the final fused image I f , the feature fusion and reconstruction process is defined as:

[0048] I f =FRRB(FFCM(F ir,f ,F vi,f )),

[0049] Among them, FFCM represents the feature fusion convolution module, FRRB represents the feature reconstruction and refinement module, and I f Represents the final fused image.

[0050] Furthermore, the Step 6 includes:

[0051] Step 6.1: The first stage trains the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including:

[0052] First, the registered infrared image and visible light image are aligned (I ir ,I vi ) is input into the feature encoder E, and the fused infrared features F are extracted respectively. ir,fand the fused visible light feature F vi,f ;

[0053] Next, the fused infrared feature F ir,f and the fused visible light feature F vi,f Input them into the reconstruction module RM respectively to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c ;

[0054] At the same time, the fused infrared feature F ir,f and the fused visible light feature F vi,f Input into the feature fusion convolution module FFCM to perform feature fusion and obtain the fused feature map;

[0055] Then, the fused feature map is input into the feature reconstruction and refinement module FRRB to generate the final fused image I f ;

[0056] Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are optimized;

[0057] The total loss function used in the first stage of training is defined as:

[0058]

[0059] in, is the fusion loss, including pixel loss Structural similarity loss and gradient loss

[0060] Fusion loss Defined as:

[0061]

[0062] Pixel loss Defined as:

[0063]

[0064] Structural similarity loss Defined as:

[0065]

[0066] Among them, SSIM calculates the structural similarity between input images.

[0067] Gradient Loss Defined as:

[0068]

[0069] in, is the Sobel gradient operator;

[0070] Reconstruction losses Used to ensure the accuracy of single-modal features, it is defined as:

[0071]

[0072] Among them, I ir,c and I vi,c are the reconstructed source infrared image and source visible light image, β, γ, and η are the weight balancing coefficients of the loss function, which are used as hyperparameters to adjust the weights of each loss term;

[0073] Step 6.2: The second stage trains the multi-view embedding module and the progressive multi-scale alignment module, specifically including:

[0074] First, the parameters of the feature encoder E, feature fusion convolution module FFCM, and feature reconstruction and refinement module FRRB after the first stage of training are frozen;

[0075] Then, the unregistered infrared image is aligned with the visible light image. Input feature encoder to extract fused unaligned infrared features and the fused visible light feature F vi,f ;

[0076] Then, the extracted features are input into the multi-view embedding module to generate pseudo deformation fields from three perspectives. Then, the pseudo deformation fields of each scale are gradually applied to the unaligned infrared features for progressive feature alignment, and the final aligned infrared features are obtained through the multi-scale multi-view progressive alignment module MSPA.

[0077] At the same time, the final aligned infrared features Input reconstruction module RM to reconstruct the aligned image

[0078] Afterwards, the final aligned infrared features and the fused visible light feature F vi,f Input to the feature fusion convolution module FFCM and the feature reconstruction and refinement module FRRB to generate a high-quality fused image after alignment and fusion

[0079] In order to ensure the accuracy of feature alignment, the reconstruction loss function is defined Computational reconstruction of aligned images With the strictly registered infrared image I ir The L1 distance between them is:

[0080]

[0081] At the same time, in order to prevent the pseudo deformation field from producing unnatural distortion, the pseudo deformation field predicted at the last scale is Introduce smoothness loss, defined as:

[0082]

[0083] In addition, the pseudo deformation field Applicable to unaligned infrared images as input Get the maximum correction result And by maximizing the correction result With the strictly registered infrared image I ir The matching loss is defined by the local normalized cross-correlation NCC:

[0084]

[0085] Finally, the total loss function of the multi-scale multi-view progressive alignment module MSPA is defined as:

[0086]

[0087] Among them, α is the weight hyperparameter of the smoothness loss term.

[0088] Furthermore, the Step 7 includes:

[0089] First, the unregistered infrared image to be processed is compared with the visible light image Input into the trained network;

[0090] Next, the feature encoder is used to extract the fused image features The infrared features are aligned through the multi-view embedding module and the progressive multi-scale alignment module;

[0091] Then, the final aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction;

[0092] Finally, the aligned and fused high-quality fused image is output

[0093] The present invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, the system comprising: a module for executing the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment.

[0094] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment is implemented.

[0095] The beneficial effects of the present invention are:

[0096] 1. Improving the feature alignment accuracy and fusion quality of unregistered infrared and visible light images: This invention introduces multi-view pseudo-deformation field modeling, combines local correlation calculation with global feature interaction, and can effectively guide the feature alignment of unregistered image pairs. The progressive multi-scale alignment module uses a progressive alignment strategy to achieve feature alignment at different scales, significantly improving the consistency of cross-modal features and the structural fidelity of the fused image, achieving high-quality fused images without the need for strict registration.

[0097] 2. Enhance the robustness and generalization ability of misaligned fusion tasks: Through the collaborative modeling of multi-view pseudo-deformation fields and multi-scale progressive optimization, the present invention reduces the impact of the prediction error of a single deformation field on the overall fusion effect and improves the fault tolerance and stability of the feature alignment process. In addition, the introduction of local normalized cross-correlation loss and deformation field smoothness constraints further enhances the generalization performance of the model under complex misaligned conditions.

[0098] 3. Optimize the training efficiency and inference effect of the fusion process: The present invention adopts a two-stage training strategy. The first stage trains the feature encoding and fusion modules on the registered data. The second stage specifically optimizes the multi-view embedding and alignment modules on the unregistered data, which improves the stability and convergence speed of the overall training and realizes the image registration and fusion tasks in an end-to-end manner. The inference stage process is simple and can efficiently complete feature extraction, alignment, fusion and image reconstruction, taking into account both performance and inference efficiency.

[0099] 4. Improve the adaptability and reliability of practical applications: The method of the present invention can adapt to the input of infrared and visible light images with different degrees of registration error, stably output high-quality fused images in a variety of scenarios, and has good practical application potential. Whether in security monitoring, night vision assistance, complex environment perception and other applications, the present invention can provide a more reliable and detailed cross-modal fusion solution, enhancing the applicability and practicality of the system.

[0100] 5. This invention fully leverages the advantages of deep learning in feature modeling and cross-modal alignment. Focusing on the key challenge of infrared and visible light image fusion under unregistered conditions, it proposes a new framework for multi-view pseudo-deformation field modeling and progressive multi-scale feature alignment, realizing end-to-end feature-level registration and fusion tasks, and solving the problems of insufficient cross-modal alignment accuracy and degraded fusion quality. It has important theoretical value and broad practical application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0101] Figure 1 A flow chart showing the structure of the method of the present invention and a comparison chart of the prior art;

[0102] Figure 2 This is a structural diagram of the progressive multi-scale alignment module of the method of the present invention;

[0103] Figure 3 This is the structure diagram of the main CNN branch UIB Block of the method of the present invention;

[0104] Figure 4 This is the structure diagram of the Transformer branch Restormer of the main body of the method of the present invention;

[0105] Figure 5 This is a performance comparison chart of the method of the present invention and different registration methods;

[0106] Figure 6 This is a performance comparison chart of the method of the present invention and different fusion methods;

[0107] Figure 7 This is the overall flow chart of the framework of the unregistered image fusion method of multi-view embedding and progressive multi-scale alignment of the present invention. DETAILED DESCRIPTION

[0108] Example 1: Figure 1-Figure 7 As shown, a method for fusion of unregistered images based on multi-view embedding and progressive multi-scale alignment includes:

[0109] Step 1: Obtain infrared image and visible light image pairs, and preprocess the input images to unify the image size and normalize them;

[0110] Furthermore, the Step 1 includes:

[0111] The preprocessing includes two stages. The infrared image and visible light image in the first stage are randomly cropped to a size of 128×128, and the images are normalized before being input into the feature encoder, feature fusion module and multi-view embedding module; the infrared image and visible light image in the second stage are scaled to a size of 400×304, and an affine transformation is applied to the infrared image, including different degrees of rotation and translation. Specifically, the rotation angle is randomly sampled in the range of -2° to 2°, and the maximum horizontal and vertical translation amplitudes are 2% of the image width and height, respectively; then, an elastic transformation is applied with a Gaussian function with a kernel size of 63×63 and a standard deviation of 32 to increase the diversity of training samples while simulating the offset in real scenes; in the preprocessing stage, the RGB visible light image is converted to the YCbCr color space, and then fused with the infrared image using the separated Y channel, and then fused with the Cb and Cr channels and converted to the RGB color space as the final fused image.

[0112] Step 2: Construct a feature encoder. The feature encoder includes a convolutional neural network branch and a Transformer branch. It extracts local detail features and global context features of infrared images and visible light images respectively, and then concatenates and fuses the extracted features to establish a feature candidate library.

[0113] Furthermore, the Step 2 includes:

[0114] Step 2.1: For the CNN branch to extract local features of the image and the Restormer branch to extract global features of the image, the input infrared image and visible light image are input into the two branches of the feature encoder E for processing to obtain preliminary extracted features, specifically including:

[0115] First, the input registered infrared image and visible light image pair (I ir ,I vi ) and unregistered infrared and visible light images Input to the shallow feature extractor SFE to extract the initial shallow features. Specifically, SFE is composed of 4 consecutive convolutional layers;

[0116] Next, the shallow features are input into the CNN branch and the Restormer branch for deep feature extraction; the CNN branch uses the UIB Block designed based on MobileNetv4, and the Restormer branch uses the Restormer module for feature extraction;

[0117] Then, multi-scale feature representations of infrared images and visible light images are obtained from the CNN branch and Restormer branch respectively;

[0118] Finally, the features obtained are recorded as and

[0119] Among them, I ir represents the strictly registered infrared image, I vi represents a visible light image, represents an unaligned infrared image, E represents a feature encoder, SFE represents a shallow feature extractor, the CNN branch adopts the UIB Block structure, and the Restormer branch adopts the Restormer module. and Represents the features extracted by different inputs through the CNN branch and the Restormer branch respectively;

[0120] Step 2.2: For the feature fusion operation under the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representation; specifically, the following steps are performed:

[0121] First, the features extracted from the infrared image and visible light image in their respective modes are concatenated.

[0122] Then, the concatenated features are passed through two convolutional blocks CB for further feature fusion and feature adjustment;

[0123] Then, the aligned and unaligned infrared features and visible light features after fusion are denoted as F ir,f 、 and F vi,f , used for subsequent generalized deformation field prediction and fusion tasks;

[0124] Among them, CB represents the convolution block operation, F ir,f represents the fused infrared features, represents the fused unaligned infrared features, F vi,f represents the fused visible light features;

[0125] Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the validity of the features. This includes:

[0126] The fused infrared feature F ir,f and the fused visible light feature F vi,f Input into the reconstruction module (RM) to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c; Among them, the reconstruction module RM consists of 6 consecutive convolutional layers, and the last convolutional layer applies the Sigmoid activation function to convert the image into a normalized space.

[0127] Step 3: Construct a multi-view embedding module, input the infrared features and visible light features in the feature candidate library into the multi-view embedding module, and predict the pseudo-deformation fields of the first, second, and third perspectives respectively through correlation calculation and feature interaction;

[0128] Furthermore, in Step 3, the operation process of the multi-view embedding module includes:

[0129] The unaligned infrared features fused in the feature candidate library and the fused visible light feature F vi,f Input into the multi-scale feature extraction network to extract the features of the final scale layer and

[0130] First, the features of the final scale layer and Perform correlation calculation within the local neighborhood to obtain the local correlation matrix R Δi,Δj (i,j), local correlation matrix R Δi,Δj (i,j) is defined as:

[0131]

[0132] Where Δi, Δj∈{-1,0,1} represents the pixel displacement amplitude in the local range; i and j represent the i-th row and j-th column of the feature map respectively; C is the total number of channels of the feature, and k is the channel index, corresponding to the k-th channel in C;

[0133] Next, the local correlation matrix R of all positions Δi,Δj (i, j) are spliced to form a complete correlation matrix R, which is defined as:

[0134] R=[R -1,-1 ,R -1,0 ,R -1,1 ,...,R 1,1 ]

[0135] Then, the complete correlation matrix R is combined with the features of the final scale layer The features are spliced and sent to the 3×3 convolution layer for feature fusion, and then further feature extraction is performed through the Restormer module; after that, the extracted features are input into the pseudo deformation field prediction network (Generalized Deformation Field, GDF) of three consecutive different branches: the first branch directly sends the extracted features to the GDF for processing to predict the pseudo deformation field of the first perspective The extracted features are then fed into the second branch UIB_Block module to further extract enhanced features in combination with channel attention. Then, another GDF is used to predict the second-perspective pseudo-deformation field. The third branch splices the pseudo deformation field of the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo deformation field of the third perspective.

[0136] In the above operation, the GDF network consists of a 3×3 convolution block with custom weights. It is to predict the pseudo deformation field of three different perspectives at the Nth scale, It is the enhanced feature obtained at the Nth scale, which will be passed to the next scale for enhanced information transmission and feature fusion.

[0137] Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction from high-level to low-level, and gradually optimize the feature alignment effect at each scale;

[0138] Furthermore, in Step 4, the operation process of the progressive multi-scale alignment module includes:

[0139] First, the N-1th layer unaligned infrared features output by the multi-scale feature extraction network are The pseudo deformation field predicted by the previous scale For the registration of the misaligned features, feature correction is performed in sequence from the highest level to the lowest level in the order of scale numbers n=(N, N-1, ..., 1);

[0140] Then, at each scale, the pseudo deformation field calculated at the previous scale is Unaligned infrared features of layer N-1 Perform progressive sampling transformation to obtain aligned infrared features

[0141] Then, the aligned infrared features Input to the Restormer module of the current scale, and the visible light features at the same scale Perform feature fusion to improve the consistency of the aligned features; then the aligned infrared features Enhanced features from the previous scale Continue to send it to the second Restormer module for feature fusion to further enhance the information transmission of features between different scales, and repeat Step 3 to obtain the pseudo deformation fields of three different perspectives at the corresponding scales.

[0142] Finally, the feature correction and fusion at all scales are completed in sequence, and the alignment effect of infrared and visible light features in multi-scale space is gradually optimized, providing a feature basis for accurate matching for subsequent fusion tasks; the progressive multi-scale feature alignment process obtains the final aligned infrared feature. The process is defined as:

[0143]

[0144] Among them, n represents the scale number, N represents the total number of scale layers, represents the unaligned infrared features of layer n, Represents the pseudo deformation field predicted at the nth layer, W is the Warp operation, and Warp represents the operation of sampling transformation based on the pseudo deformation field. represents the visible light characteristics of the nth layer, represents the aligned infrared features after applying the pseudo deformation field transformation, is the final aligned infrared signature.

[0145] Step 5: Construct a feature fusion module, input the progressively aligned infrared features and visible light features into the feature fusion convolution module, perform feature fusion through convolution operation, and generate a fused feature map; input the fused feature map into the feature reconstruction and refinement module, and generate the final fused image through convolution and activation operations;

[0146] Furthermore, in Step 5, the operation process of the feature fusion module includes:

[0147] First, the aligned fused infrared features F ir,f and the fused visible light feature F vi,f Input to the Feature Fusion Convolutional Module (FFCM), perform feature fusion through convolution operation, and generate a fused feature map;

[0148] Next, the fused feature map is input into the Feature Reconstruction and Refinement Block (FRRB). In the FRRB module, the feature representation capability is first enhanced by the Channel Attention module (CA). Then, the features are further extracted by a convolution block containing a 3×3 convolution layer, a batch normalization (BN) layer, and a LeakyReLU (LReLU) activation function. Finally, a 1×1 convolution layer is used with a Sigmoid activation function to output the final fused image I. f , the feature fusion and reconstruction process is defined as:

[0149] I f =FRRB(FFCM(F ir,f ,F vi,f )),

[0150] Among them, FFCM represents the feature fusion convolution module, FRRB represents the feature reconstruction and refinement module, and I f Represents the final fused image.

[0151] Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module.

[0152] Furthermore, the Step 6 includes:

[0153] Step 6.1: The first stage trains the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including:

[0154] First, the registered infrared image and visible light image are aligned (I ir ,I vi ) is input into the feature encoder E, and the fused infrared features F are extracted respectively. ir,f and the fused visible light feature F vi,f ;

[0155] Next, the fused infrared feature F ir,f and the fused visible light feature F vi,f They are respectively input into the reconstruction module (RM) to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c ;

[0156] At the same time, the fused infrared feature F ir,f and the fused visible light feature Fvi,f Input into the feature fusion convolution module FFCM to perform feature fusion and obtain the fused feature map;

[0157] Then, the fused feature map is input into the feature reconstruction and refinement module FRRB to generate the final fused image I f ;

[0158] Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are optimized;

[0159] The total loss function used in the first stage of training is defined as:

[0160]

[0161] in, is the fusion loss, including pixel loss Structural similarity loss and gradient loss

[0162] Fusion loss Defined as:

[0163]

[0164] Pixel loss Defined as:

[0165]

[0166] Structural similarity loss Defined as:

[0167]

[0168] Among them, SSIM calculates the structural similarity between input images.

[0169] Gradient Loss Defined as:

[0170]

[0171] in, is the Sobel gradient operator;

[0172] Reconstruction losses Used to ensure the accuracy of single-modal features, it is defined as:

[0173]

[0174] Among them, I ir,c and Ivi,c are the reconstructed source infrared image and source visible light image, β, γ, and η are the weight balancing coefficients of the loss function, which are used as hyperparameters to adjust the weights of each loss term;

[0175] Step 6.2: The second stage trains the multi-view embedding module and the progressive multi-scale alignment module, specifically including:

[0176] First, the parameters of the feature encoder E, feature fusion convolution module FFCM, and feature reconstruction and refinement module FRRB after the first stage of training are frozen; only the multi-scale multi-view progressive alignment module (MSPA) is trained;

[0177] Then, the unregistered infrared image is aligned with the visible light image. Input feature encoder to extract fused unaligned infrared features and the fused visible light feature F vi,f ;

[0178] Then, the extracted features are input into the multi-view embedding module to generate pseudo deformation fields from three perspectives. Then, the feature transfer is performed step by step, and the pseudo deformation fields of each scale are gradually applied to the unaligned infrared features for progressive feature alignment. The final aligned infrared features are obtained through the multi-scale multi-view progressive alignment module MSPA.

[0179] At the same time, the final aligned infrared features Input reconstruction module RM to reconstruct the aligned image

[0180] Afterwards, the final aligned infrared features and the fused visible light feature F vi,f Input to the feature fusion convolution module FFCM and the feature reconstruction and refinement module FRRB to generate a high-quality fused image after alignment and fusion

[0181] In order to ensure the accuracy of feature alignment, the reconstruction loss function is defined Computational reconstruction of aligned images With the strictly registered infrared image I ir The L1 distance between them is:

[0182]

[0183] At the same time, in order to prevent the pseudo deformation field from producing unnatural distortion, the pseudo deformation field predicted at the last scale is Introduce smoothness loss, defined as:

[0184]

[0185] In addition, the pseudo deformation field Applicable to unaligned infrared images as input Get the maximum correction result And by maximizing the correction result With the strictly registered infrared image I ir The matching loss is defined by the local normalized cross-correlation NCC:

[0186]

[0187] Finally, the total loss function of the multi-scale multi-view progressive alignment module MSPA is defined as:

[0188]

[0189] Among them, α is the weight hyperparameter of the smoothness loss term.

[0190] Step 7: Input the unregistered infrared and visible light images to be processed into the trained network and output the aligned and fused high-quality fused images.

[0191] Furthermore, the Step 7 includes:

[0192] First, the unregistered infrared image to be processed is compared with the visible light image (I ir ,I vi ) is input into the trained network;

[0193] Next, the feature encoder is used to extract the fused image features The infrared features are aligned through the multi-view embedding module and the progressive multi-scale alignment module;

[0194] Then, the final aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction;

[0195] Finally, the aligned and fused high-quality fused image is output

[0196] At the same time, when the multi-scale multi-view progressive alignment module (MSPA) is not activated, the model can input the aligned infrared and visible light image pairs (I ir ,I vi ), and obtain the high-quality fused image I under the aligned scene f .

[0197] The present invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized in that the system comprises:

[0198] Image acquisition and preprocessing module, used to acquire infrared image and visible light image pairs, and preprocess the input images, unify the image size and normalize them;

[0199] The feature candidate library establishment module is used to build a feature encoder. The feature encoder includes a convolutional neural network branch and a Transformer branch. It extracts local detail features and global context features of infrared images and visible light images respectively, and then splices and fuses the extracted features to establish a feature candidate library.

[0200] The multi-view embedding module is used to input the infrared features and visible light features in the feature candidate library into the multi-view embedding module, and predict the pseudo deformation fields of the first view, second view, and third view respectively through correlation calculation and feature interaction;

[0201] Progressive multi-scale alignment module, which applies the pseudo-deformation field of each scale to the infrared features, performs feature correction from high-level to low-level, and gradually optimizes the feature alignment effect at each scale;

[0202] The feature fusion module is used to input the infrared features and visible light features after progressive alignment into the feature fusion convolution module, perform feature fusion through convolution operation, and generate a fused feature map; the fused feature map is input into the feature reconstruction and refinement module, and the final fused image is generated through convolution and activation operations;

[0203] A training module that uses a two-stage training approach. In the first stage, the feature encoder and feature fusion module are trained using registered image pairs. In the second stage, the multi-view embedding module and the progressive multi-scale alignment module are trained using unregistered image pairs.

[0204] The generation module is used to input the unregistered infrared and visible light images to be processed into the trained network and output a high-quality fused image after alignment and fusion.

[0205] Furthermore, the present invention also provides an unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized in that the system comprises:

[0206] The preprocessing module preprocesses the infrared image and visible light image pairs, performs channel separation, random cropping, scaling, affine and elastic transformation, and normalization operations on the images to meet the model input requirements.

[0207] The shallow feature extraction network performs preliminary feature extraction on the input infrared and visible light image pairs, converts low-dimensional images into high-dimensional features, mines shallow features in the image, and provides basic feature requirements for subsequent feature extraction networks.

[0208] The backbone feature extraction network includes a sub-network composed of CNN for extracting local features and a sub-network composed of Restormer for extracting global features. It is used to feed shallow features into the corresponding sub-network to further mine the global and local information in the image features.

[0209] A multi-scale and multi-view progressive alignment network is used to extract features at different scales of the image. A CNN sub-network and a Restormer self-network are then used to predict pseudo-deformation fields at multiple viewpoints at different scales. The pseudo-deformation fields are then applied to the offset infrared image features to obtain registered and aligned infrared image features. The features are then transferred layer by layer to predict the final multi-view pseudo-deformation field, which is then applied to the original offset infrared image features to obtain the final scale-aligned infrared image features.

[0210] The reconstruction network can reconstruct the input features from high dimensions into corresponding modal images through the convolution module, and can urge the network to retain the original image semantic information as much as possible.

[0211] Feature fusion network, which can perform cross-modal fusion of the offset infrared image features and visible light image features, fully explore the feature information in different modalities, and effectively fuse the two modal features, thereby improving the information flow between features.

[0212] A fusion image reconstruction network is used to reconstruct images based on the fusion features. The network can obtain the final fusion image based on the fusion features. The fusion image contains the detail information in the visible light image and the salient targets in the infrared image.

[0213] The training module, through the design of different loss functions, can train sub-networks in a two-stage manner, including: shallow feature extraction network, backbone feature extraction network, multi-scale and multi-view progressive alignment network, reconstruction network, feature fusion network, and fused image reconstruction network.

[0214] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment is implemented.

[0215] This paper innovatively introduces a multi-view and multi-scale feature-level pseudo-deformation field prediction method. First, based on global and local feature extraction, fusion features of the corresponding modalities are obtained. Next, the features are interacted with the global context through local correlation to generate pseudo-deformation fields of different viewpoints to collaboratively guide feature-level alignment. At the same time, a multi-scale progressive alignment strategy is adopted to gradually refine and correct offset features from different scales, which can significantly reduce the dependence on and accuracy requirements of a single deformation field. A two-stage training paradigm is used to improve the effectiveness of feature-level registration. On this basis, through the feature fusion module and feature reconstruction module, the full integration and fine-grained recovery of infrared and visible light information are achieved, and the final output is a high-quality fused image with rich details and consistent structure.

[0216] To validate the effectiveness of our method, we constructed training data based on the RoadScene dataset. The RoadScene dataset contains 221 pairs of registered infrared and visible light images, of which 200 pairs were randomly selected as the training set, and the remaining 21 pairs were used as the test set. To further evaluate the robustness and generalization ability of our method, 50 pairs of infrared and visible light image pairs were randomly selected from the M3FD dataset and 40 pairs of infrared and visible light image pairs were randomly selected from the MSRS dataset as additional test sets. During training and testing, to simulate registration errors that may occur in real-world applications, we generated unregistered samples using affine transformations and elastic deformations. The affine transformations included random rotations and translations with rotation angles ranging from -2° to 2°, and horizontal and vertical translations with a maximum amplitude of 2% of the image width and height. Elastic deformations used a Gaussian function with a kernel size of 63×63 and a standard deviation of 32 to generate random local deformations, thereby simulating nonlinear local distortions. Our method was developed based on the PyTorch framework and trained and tested on hardware equipped with an NVIDIA GeForce RTX 4090 graphics card. Model training consists of two phases. In the first phase, the feature encoder (FE) and feature decoder (FD) are trained using registered image pairs as input. In this phase, the input images are randomly cropped to 128×128 pixels, the batch size is set to 10, the initial learning rate is set to 1e-2, and cosine annealing is used to gradually decay the learning rate. Training is performed for 2000 epochs. In the second phase, the unregistered image pairs are used as input, and only the parameters of the progressive multi-scale alignment module (MSPA) are optimized, while the remaining modules are frozen. In this phase, the input images are uniformly resized to 400×304 pixels, the batch size is set to 4, the initial learning rate is set to 1e-4, and cosine annealing is continued for 4000 epochs, with each epoch consisting of 50 iterations. During training, the number of training samples is effectively expanded by continuously applying different affine and elastic transformations, ultimately exceeding 800,000. The training strategy of the present invention not only effectively improves the feature alignment ability and fusion performance of the model under unregistered conditions, but also significantly enhances the robustness and generalization ability of the model in different scenarios and degradation conditions.

[0217] Further explanation, the present invention uses mutual information (MI), cross entropy (CE), visual information fidelity (VIF), gradient-based fusion evaluation index (Q AB / F ), evaluation index based on edge features (Q CB ) and the evaluation index based on texture features (Q CV) are used to evaluate the quality of the fused image. Among them, mutual information (MI) is used to measure the amount of source image information contained in the fused image. The higher the MI value, the more effective information of the source image the fused image retains. Cross entropy (CE) is mainly used to reflect the consistency of information distribution between the fused image and the source image. The lower the CE value, the better the fusion effect. Visual information fidelity (VIF) evaluates the degree to which the fused image retains the source image features at the visual perception level. The higher the VIF value, the closer the fused image is to the source image in visual perception. Gradient-based fusion evaluation index (Q AB / F ) focuses on the preservation of the gradient information of the fused image, which can effectively reflect the degree of preservation of image edge details and clarity. AB / F The higher the value, the better the preservation effect of the edge structure of the fused image; the evaluation index based on edge features (Q CB ) is used to measure the correlation between the edge features in the fused image and the source image, Q CB The higher the value, the better the fused image can maintain the edge details of the source image; the evaluation index based on texture features (Q CV ) reflects the integrity and richness of the texture features in the fused image, Q CV The higher the value, the better the fused image's performance in texture detail. Through the comprehensive evaluation of these six indicators, this paper can comprehensively and objectively assess the performance of the fused image in terms of information retention, structural consistency, and visual perception quality, ensuring that the fusion result not only preserves high-quality details but also meets the perception requirements of the human visual system.

[0218] Table 1 Quantitative comparison experimental results of registration and fusion methods on multiple datasets

[0219]

[0220]

[0221] Furthermore, the proposed image fusion method was compared with existing advanced methods such as UMF-CMGR, SuperFusion, MURF, SemLA, RFIVF, IMF, and IVFWSR on the RoadScene, M3FD, and MSRS datasets. The results are shown in Table 1. The proposed method achieved excellent performance across all metrics on the three datasets, particularly in terms of information retention, structure preservation, and visual quality, significantly outperforming the compared methods.

[0222] To further verify the effectiveness of this method in the misaligned scene, the aligned infrared features are The reconstructed aligned image is obtained through the feature reconstruction module RM After that, the strictly aligned infrared image I ir Calculate various indicators used to evaluate the image alignment effect. Among them, the indicators include: MI, which measures the consistency of shared content between two images by the commonality of information, MS-SSIM, which evaluates perceptual similarity from the perspective of multi-scale structure, and NCC (normalized cross-correlation coefficient), which describes the linear correlation of image intensities. The higher the value of the above indicators, the better the alignment effect of image registration performance. The specific experimental results are as follows: Figure 5 shown.

[0223] The qualitative experimental results compared with the above methods are as follows Figure 6 As shown in the figure, the experimental results of the present invention can achieve higher-quality fusion effects and effectively alleviate the offset and visual misalignment effects in the image. These experimental results fully verify the effectiveness of the method of the present invention in different datasets and different scene conditions, showing good fusion performance and excellent cross-dataset generalization ability, further demonstrating the application potential and practical value of the present invention in the task of fusion of unregistered infrared and visible light images.

[0224] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. An unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment, characterized by: The method comprises: Step 1: Obtain infrared image and visible light image pairs, and preprocess the input images to unify the image size and normalize them; Step 2: Construct a feature encoder. The feature encoder includes a convolutional neural network branch and a Transformer branch. It extracts local detail features and global context features of infrared images and visible light images respectively, and then concatenates and fuses the extracted features to establish a feature candidate library. Step 3: Construct a multi-view embedding module, input the infrared features and visible light features in the feature candidate library into the multi-view embedding module, and predict the pseudo-deformation fields of the first, second, and third perspectives respectively through correlation calculation and feature interaction; Step 4: Construct a progressive multi-scale alignment module, apply the pseudo-deformation field of each scale to the infrared features, perform feature correction from high-level to low-level, and gradually optimize the feature alignment effect at each scale; Step 5: Construct a feature fusion module, input the progressively aligned infrared features and visible light features into the feature fusion convolution module, perform feature fusion through convolution operation, and generate a fused feature map; input the fused feature map into the feature reconstruction and refinement module, and generate the final fused image through convolution and activation operations; Step 6: A two-stage training method is adopted. In the first stage, the registered image pairs are used to train the feature encoder and feature fusion module. In the second stage, the unregistered image pairs are used to train the multi-view embedding module and the progressive multi-scale alignment module. Step 7: Input the unregistered infrared and visible light images to be processed into the trained network and output the aligned and fused high-quality fused images.

2. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 1 includes: The preprocessing consists of two stages. In the first stage, the infrared images and visible light images are randomly cropped to a size of 128×128, and the images are normalized before being input into the feature encoder, feature fusion module, and multi-view embedding module. In the second stage, the infrared images and visible light images are scaled to a size of 400×304, and an affine transformation is applied to the infrared images, including different degrees of rotation and translation. Then, an elastic transformation is applied with a Gaussian function with a kernel size of 63×63 and a standard deviation of 32 to increase the diversity of training samples while simulating the offset conditions in real scenes.

3. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 2 includes: Step 2.1: For the feature extraction operation of the CNN branch and the Restormer branch, the input infrared image and visible light image are respectively input into the two branches of the feature encoder E for processing to obtain the preliminary extracted features, specifically including: First, the input registered infrared image and visible light image pair (I ir ,I vi ) and unregistered infrared and visible light images Input into the shallow feature extractor SFE to extract the initial shallow features; Then, the shallow features are input into the CNN branch and Restormer branch respectively for deep feature extraction; Then, multi-scale feature representations of infrared images and visible light images are obtained from the CNN branch and Restormer branch respectively; Finally, the features obtained are recorded as Among them, I ir represents the strictly registered infrared image, I vi represents a visible light image, represents an unaligned infrared image, E represents a feature encoder, SFE represents a shallow feature extractor, the CNN branch adopts the UIB Block structure, and the Restormer branch adopts the Restormer module. and Represents the features extracted by different inputs through the CNN branch and the Restormer branch respectively; Step 2.2: For the feature fusion operation under the same modality, the features extracted by the CNN branch and the Restormer branch are fused to obtain high-quality feature representation; specifically, the following steps are performed: First, the features extracted from the infrared image and visible light image in their respective modalities are spliced separately; Then, the spliced features are passed through two convolution blocks CB for further feature fusion and feature adjustment; Then, the aligned and unaligned infrared features and visible light features after fusion are denoted as F ir,f 、 and F vi,f , used for subsequent generalized deformation field prediction and fusion tasks; Among them, CB represents the convolution block operation, F ir,f represents the fused infrared features, represents the fused unaligned infrared features, F vi,f represents the fused visible light features; Step 2.3: For feature quality assurance, the fused features are input into the reconstruction module for image reconstruction to ensure the validity of the features. This includes: The fused infrared feature F ir,f and the fused visible light feature F vi,f Input into the reconstruction module RM to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c .

4. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 3, the operation process of the multi-view embedding module includes: The unaligned infrared features fused in the feature candidate library and the fused visible light feature F vi,f Input into the multi-scale feature extraction network to extract the features of the final scale layer and First, the features of the final scale layer and Perform correlation calculation within the local neighborhood to obtain the local correlation matrix R Δi,Δj (i,j), local correlation matrix R Δi,Δj (i,j) is defined as: Where Δi, Δj∈{-1,0,1} represents the pixel displacement amplitude in the local range; i and j represent the i-th row and j-th column of the feature map respectively; C is the total number of channels of the feature, and k is the channel index, corresponding to the k-th channel in C; Next, the local correlation matrix R of all positions Δi,Δj (i, j) are spliced to form a complete correlation matrix R, which is defined as: R=[R -1,-1 ,R -1,0 ,R -1,1 ,...,R 1,1 ] Then, the complete correlation matrix R is combined with the features of the final scale layer The features are spliced and sent to the 3×3 convolution layer and the Restormer module for further feature extraction. After that, the extracted features are input into the pseudo deformation field prediction network GDF of three consecutive different branches: the first branch directly sends the extracted features to the GDF for processing to predict the pseudo deformation field of the first perspective. The extracted features are then fed into the second branch UIB_Block module to further extract enhanced features in combination with channel attention. Then, another GDF is used to predict the second-perspective pseudo-deformation field. The third branch splices the pseudo deformation field of the first and second perspectives and inputs them into the third GDF prediction network to predict the pseudo deformation field of the third perspective. In the above operation, the GDF network consists of a 3×3 convolution block with custom weights. It is to predict the pseudo deformation field of three different perspectives at the Nth scale, It is the enhanced feature obtained at the Nth scale, which will be passed to the next scale for enhanced information transmission and feature fusion.

5. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 4, the operation process of the progressive multi-scale alignment module includes: First, the N-1th layer unaligned infrared features output by the multi-scale feature extraction network are The pseudo deformation field predicted by the previous scale For the registration of the misaligned features, feature correction is performed in sequence from the highest level to the lowest level in the order of scale numbers n=(N, N-1, ..., 1); Then, at each scale, the pseudo deformation field calculated at the previous scale is Unaligned infrared features of layer N-1 Perform progressive sampling transformation to obtain aligned infrared features Then, the aligned infrared features Input to the Restormer module of the current scale, and the visible light features at the same scale Perform feature fusion to improve the consistency of the aligned features; then the aligned infrared features Enhanced features from the previous scale Continue to send it to the second Restormer module for feature fusion to further enhance the information transmission of features between different scales, and repeat Step 3 to obtain the pseudo deformation fields of three different perspectives at the corresponding scales. Finally, the feature correction and fusion at all scales are completed in sequence, and the alignment effect of infrared and visible light features in multi-scale space is gradually optimized, providing a feature basis for accurate matching for subsequent fusion tasks; the progressive multi-scale feature alignment process obtains the final aligned infrared feature. The process is defined as: Among them, n represents the scale number, N represents the total number of scale layers, represents the unaligned infrared features of layer n, Represents the pseudo deformation field predicted at the nth layer, W is the Warp operation, and Warp represents the operation of sampling transformation based on the pseudo deformation field. represents the visible light characteristics of the nth layer, represents the aligned infrared features after applying the pseudo deformation field transformation, is the final aligned infrared signature.

6. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: In Step 5, the operation process of the feature fusion module includes: First, the aligned fused infrared features F ir,f and the fused visible light feature F vi,f Input to the feature fusion convolution module FFCM, perform feature fusion through convolution operation, and generate a fused feature map; Next, the fused feature map is input into the feature reconstruction and refinement module FRRB. In the FRRB module, the feature representation capability is first enhanced by the channel attention module CA, and then the features are further extracted by a convolution block containing a 3×3 convolution layer, a batch normalization BN layer and a LeakyReLU activation function. Finally, a 1×1 convolution layer is used with a Sigmoid activation function to output the final fused image I f , the feature fusion and reconstruction process is defined as: I f =FRRB(FFCM(F ir,f ,F vi,f )), Among them, FFCM represents the feature fusion convolution module, FRRB represents the feature reconstruction and refinement module, and I f Represents the final fused image.

7. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 6 includes: Step 6.1: The first stage trains the feature encoder, feature fusion convolution module, and feature reconstruction and refinement module, specifically including: First, the registered infrared image and visible light image are aligned (I ir ,I vi ) is input into the feature encoder E, and the fused infrared features F are extracted respectively. ir,f and the fused visible light feature F vi,f ; Next, the fused infrared feature F ir,f and the fused visible light feature F vi,f Input them into the reconstruction module RM respectively to reconstruct the corresponding source infrared image I ir,c and source visible light image I vi,c ; At the same time, the fused infrared feature F ir,f and the fused visible light feature F vi,f Input into the feature fusion convolution module FFCM to perform feature fusion and obtain the fused feature map; Then, the fused feature map is input into the feature reconstruction and refinement module FRRB to generate the final fused image I f ; Finally, by minimizing the reconstruction error and fusion error between the fused image and the original input image, the parameters of the feature encoder E, the feature fusion convolution module FFCM, and the feature reconstruction and refinement module FRRB are optimized; The total loss function used in the first stage of training is defined as: in, is the fusion loss, including pixel loss Structural similarity loss and gradient loss Fusion loss Defined as: Pixel loss Defined as: Structural similarity loss Defined as: Among them, SSIM calculates the structural similarity between input images. Gradient Loss Defined as: in, is the Sobel gradient operator; Reconstruction losses Used to ensure the accuracy of single-modal features, it is defined as: Among them, I ir,c and I vi,c are the reconstructed source infrared image and source visible light image, β, γ, and η are the weight balancing coefficients of the loss function, which are used as hyperparameters to adjust the weights of each loss term; Step 6.2: The second stage trains the multi-view embedding module and the progressive multi-scale alignment module, specifically including: First, the parameters of the feature encoder E, feature fusion convolution module FFCM, and feature reconstruction and refinement module FRRB after the first stage of training are frozen; Then, the unregistered infrared image is aligned with the visible light image. Input feature encoder to extract fused unaligned infrared features and the fused visible light feature F vi,f ; Then, the extracted features are input into the multi-view embedding module to generate pseudo deformation fields from three perspectives. Then, the pseudo deformation fields of each scale are gradually applied to the unaligned infrared features for progressive feature alignment, and the final aligned infrared features are obtained through the multi-scale multi-view progressive alignment module MSPA. At the same time, the final aligned infrared features Input reconstruction module RM to reconstruct the aligned image Afterwards, the final aligned infrared features and the fused visible light feature F vi,f Input to the feature fusion convolution module FFCM and the feature reconstruction and refinement module FRRB to generate a high-quality fused image after alignment and fusion In order to ensure the accuracy of feature alignment, the reconstruction loss function is defined Computational reconstruction of aligned images With the strictly registered infrared image I ir The L1 distance between them is: At the same time, in order to prevent the pseudo deformation field from producing unnatural distortion, the pseudo deformation field predicted at the last scale is Introduce smoothness loss, defined as: In addition, the pseudo deformation field Applicable to unaligned infrared images as input Get the maximum correction result And by maximizing the correction result With the strictly registered infrared image I ir The matching loss is defined by the local normalized cross-correlation NCC: Finally, the total loss function of the multi-scale multi-view progressive alignment module MSPA is defined as: Among them, α is the weight hyperparameter of the smoothness loss term.

8. The unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to claim 1, characterized in that: Step 7 includes: First, the unregistered infrared image to be processed is compared with the visible light image Input into the trained network; Next, the feature encoder is used to extract the fused image features The infrared features are aligned through the multi-view embedding module and the progressive multi-scale alignment module; Then, the final aligned infrared features are combined with the fused visible light features. Perform feature fusion and reconstruction; Finally, the aligned and fused high-quality fused image is output 9. Unregistered image fusion system based on multi-view embedding and progressive multi-scale alignment, characterized by: The system includes: a module for executing the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the unregistered image fusion method based on multi-view embedding and progressive multi-scale alignment as described in any one of claims 1 to 8 is implemented.

Images