Infrared and visible image registration and fusion method based on physical heuristic feature decoupling

Abstract

The present application relates to an infrared and visible light image registration and fusion method based on physical heuristic feature decoupling, which solves the problems of low registration accuracy, serious feature dislocation, ghosting, edge blur and texture loss in the fusion results of infrared and visible light image fusion technology in the face of large-scale geometric mismatch and nonlinear deformation scenes compared with the prior art. The present application comprises the following steps: acquisition and preprocessing of infrared and visible light images; constructing a physical heuristic feature decoupling mechanism for cross-modal image processing; constructing a two-stage cascaded registration network architecture based on decoupled features; constructing a layered fusion network architecture based on painting logic; training of the registration and fusion model; acquisition and preprocessing of the image to be registered and fused; obtaining the registration and fusion image result. The present application realizes high-fidelity, ghost-free fusion in large-scale mismatch scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multimodal image processing technology, specifically to a method for infrared and visible light image registration and fusion based on physically heuristic feature decoupling. Background Technology

[0002] With the development of multimodal sensing technology, infrared and visible light image fusion is widely used in fields such as autonomous driving and security monitoring. However, due to sensor installation deviations, hardware jitter, and differences in field of view, the original image pairs often exhibit complex affine offsets and nonlinear elastic deformations. This geometric mismatch leads to severe ghosting artifacts and edge blurring in the fusion results, limiting the reliability of the system.

[0003] Existing processing methods have significant limitations: style transfer-based methods rely on image translation, which easily introduces geometric distortion and has high computational cost; latent space-based methods are often "black boxes" that lack physical interpretability and have limited ability to capture large displacement deviations, making it difficult to effectively deal with large-scale nonlinear distortions in real-world scenarios.

[0004] Therefore, how to construct a physically interpretable mechanism to achieve accurate registration and high-fidelity fusion under large-scale mismatch is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing infrared and visible light image fusion technologies, such as low registration accuracy, severe feature misalignment, and common issues like ghosting, edge blurring, and texture loss in the fusion results, when faced with large-scale geometric mismatches and nonlinear deformation scenarios. This invention provides a physically heuristic feature decoupling-based infrared and visible light image registration and fusion method to solve these problems.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows:

[0007] An infrared and visible light image registration and fusion method based on physically heuristic feature decoupling includes the following steps:

[0008] 11) Acquisition and preprocessing of infrared and visible light images: Acquire registered infrared and visible light image source data and adjust them to a specified size. Keep the infrared image as a reference. Apply random global affine transformation and local elastic deformation to the visible light image through spatial geometric transformation technology to generate cross-modal image training pairs containing unregistered images. At the same time, calculate and generate the inverse affine transformation matrix corresponding to the artificial deformation, the inverse deformation field of registration, and the true registration effective mask of the effective area of ​​the registered image.

[0009] 12) Construct a physically heuristic feature decoupling mechanism for cross-modal image processing;

[0010] 13) Construct a two-stage cascaded registration network architecture based on decoupling features;

[0011] 14) Construct a hierarchical fusion network architecture based on painting logic;

[0012] 15) Training of the registration and fusion model: A phased training and teacher-student knowledge distillation strategy is adopted for training. First, the iterative global affine regression network is trained for large-scale affine coarse registration. Then, the iterative global affine regression network is frozen, and the elastic registration network is trained for local elastic fine registration. Finally, the iterative global affine regression network and the elastic registration network are frozen, and the registered source images are used as teacher labels to train the hierarchical fusion network for teacher-student knowledge distillation.

[0013] 16) Acquisition and preprocessing of the image to be registered and fused: Acquire the unregistered infrared and visible light images to be processed in the actual scene, and use the mechanism of step 12) to perform physical heuristic feature decoupling;

[0014] 17) Obtaining the registration and fusion image results: The preprocessed structural features to be registered are input into a two-stage cascaded registration network, which outputs the predicted joint dense spatial displacement field and the effective registration mask. The joint dense spatial displacement field is applied to the visible light component features to be registered to achieve registration. Finally, the registered visible light component features, infrared component features and the effective mask are input into a hierarchical fusion network. Feature-level reconstruction is performed in the effective area indicated by the mask, and the final high-quality registration and fusion image is output.

[0015] The constructed physical heuristic feature decoupling mechanism for cross-modal image processing is as follows: Based on imaging physics, the input cross-modal image training pair is explicitly decomposed into illumination feature components carrying illumination information, structural feature components carrying geometric contours, and texture feature components carrying modality-specific details. Simultaneously, the decoupling process also generates enhanced structural features, collectively providing a unified feature benchmark for cross-modal registration and fusion. This includes the following steps:

[0016] 21) Convolve the cross-modal image training pair using a large-scale Gaussian low-pass filter to extract the low-frequency illumination component L, and subtract the low-frequency illumination component L from the original image to obtain the high-frequency detail component H.

[0017] 22) An L0 norm smoothing operator is introduced to optimize the high-frequency detail component H. By minimizing the non-zero pixel statistics of the gradient, thermal diffusion noise and low-contrast textures are filtered out. Combined with contrast-limited adaptive histogram equalization, an enhanced structural feature component S is generated. reg ;

[0018] 23) With enhanced structural features Sreg To guide the image, guided filtering techniques are used to decompose the high-frequency detail component H to obtain structural features S. fuse and texture feature T;

[0019] 24) After decoupling, the cross-modal image training pairs yielded eight feature components, as follows: Visible light image enhanced structural features S reg,vi Infrared image enhanced structural features S reg,ir Visible light image structural features S fuse,vi Infrared image structural features S fuse,ir Visible light image texture features T vi Infrared image texture features T ir Visible light image illumination features L vi Infrared image illumination features L ir .

[0020] The two-stage cascaded registration network architecture based on decoupling features is constructed as follows: A cascaded deep learning architecture is adopted, utilizing the enhanced structural features after decoupling to predict the affine matrix through an iterative global affine regression network based on U-Net, correcting large-scale linear offsets between images; an elastic registration network based on RAFT is introduced to iteratively refine the displacement field in a single-scale feature space, capturing and correcting local nonlinear elastic deformations; it includes the following steps:

[0021] 31) The two-stage cascaded registration network architecture is set up, including the first stage of an iterative global affine regression network based on U-Net and the second stage of an elastic registration network based on RAFT.

[0022] 32) Setting up an iterative global affine regression network based on U-Net:

[0023] 321) Initial State Setting and Dynamic Joint Feature Construction: Set the input of the affine regression network to S reg,vi and S reg,ir The initial cumulative affine transformation matrix is ​​the identity matrix; in each iteration, the cumulative affine transformation matrix from the previous iteration step is first applied to S. reg,vi A spatial transformation is performed to obtain the visible light features in the current aligned state; subsequently, S is extracted respectively. reg,ir The Sobel spatial gradient of the visible light features in the current aligned state is calculated and averaged to generate a structural gradient guidance map; finally, S... reg,ir The visible light features in the current alignment state and the structural gradient guidance Figure 3 The components are concatenated along the channel dimension and used as the joint input to the iterative global affine regression network of the current iteration step.

[0024] 322) Residual parameter regression based on U-Net bottleneck layer: Modify the lightweight U-Net as the feature extractor, discard the image reconstruction branch of the decoder, feed the joint input into the encoder for downsampling, extract the bottleneck layer features of the deep global geometric context, and generate a two-dimensional bottleneck layer feature map; then, use an adaptive global average pooling layer to compress the two-dimensional bottleneck layer feature map into a one-dimensional feature vector, and predict the affine transformation residual increment parameters through a linear regression head initialized to zero weights;

[0025] 323) Homogeneous Matrix Chain Multiplication and Iterative Correction: In each iteration, the predicted affine transformation residual increment parameters are analyzed as rotation scaling increments and translation increments, and constructed as the increment affine matrix for the current iteration step; the increment affine matrix and the cumulative affine transformation matrix of the previous iteration step are both converted into 3×3 homogeneous transformation matrices, and composite updates are completed through homogeneous matrix multiplication to obtain the cumulative affine transformation matrix for the current iteration step; the latest cumulative matrix is ​​output to the next iteration for visible light feature resampling in step 321); through a set number of multi-step recursive iterations, the true deformation is gradually approximated, and finally the global affine transformation matrix M that eliminates large-scale linear offset is output;

[0026] 33) Setting up a RAFT-based flexible registration network:

[0027] 331) Modality-independent feature extraction: Set the input of the elastic registration network to S reg,ir And S after affine transformation matrix M reg,vi ;

[0028] The flexible registration network is configured to include a non-shared feature encoder, a MIND-SSC module, and a shared feature encoder.

[0029] The non-shared feature encoder comprises downsampling convolutional layers with large-scale convolutional kernels, instance normalization layers, and residual convolutional blocks, used to process the input S... reg,ir And S after global affine transformation matrix M reg,vi Initial spatial dimensionality reduction and independent feature mapping are performed to obtain single-channel features for both modalities;

[0030] The single-channel features of the two modalities are input into the MIND-SSC module, and the structural self-similarity distance in the local neighborhood of the pixel is calculated based on multiple preset spatial offset directions to generate a multi-channel structural descriptor.

[0031] The shared feature encoder includes cascaded downsampled residual convolutional blocks for further spatial dimensionality reduction and deep common feature extraction of the structural descriptor. Two 1×1 convolutional layers are connected in parallel at the end of the shared feature encoder to output infrared matching features and visible light matching features for calculating similarity, as well as contextual features for guiding the update of the hidden layer state.

[0032] 332) Initialization of coordinate grid and dynamic local cost volume: Before the start of the loop iteration, a standard two-dimensional pixel grid with the same dimension as the input feature space is generated as the initial absolute coordinate grid; the infrared matching feature and visible light matching feature output from step 331) are set to be received and pre-smoothed by two-dimensional Gaussian filtering to suppress multimodal feature-level noise; in each iteration, a local search grid is generated within the set local search radius with the absolute coordinate grid of the current iteration step as the sampling center; the visible light matching feature is spatially resampled using the local search grid, and the L2 squared distance between the resampled feature and the infrared matching feature in the channel dimension is calculated to dynamically construct a single-scale local cost volume, and the local cost volume is output as an error signal;

[0033] 333) ConvGRU iterative loop and displacement field update:

[0034] An update block is defined, which includes a feature fusion convolutional layer, a gated recurrent unit, and a flow field regression head. In each iteration, the absolute coordinate grid of the current iteration step is first subtracted from the initial absolute coordinate grid to calculate the dense displacement field of the current iteration step. Then, the feature fusion convolutional layer is used to perform convolutional dimensionality reduction on the local cost convolution in step 332) and the dense displacement field of the current iteration step, and concatenated with the context features output in step 331) in the channel dimension as the joint input of ConvGRU. ConvGRU uses the internal convolutional update gate and reset gate to infer the error gradient in combination with the hidden state of the previous round and outputs the updated hidden state. The hidden state is fed into the flow field regression head containing multiple convolutions to output the residual update amount of the dense displacement field. The residual update amount is superimposed on the absolute coordinate grid of the current iteration step to obtain the updated absolute coordinate grid, and it is returned to step 332) as the sampling center of the next iteration, forming a closed loop, gradually approximating the real nonlinear elastic deformation.

[0035] 334) Displacement field upsampling and smoothing mechanism: After the loop iteration is completed, the updated coordinate grid of the last output in step 333) is subtracted from the initial absolute coordinate grid to obtain a low-resolution dense displacement field. In order to adapt to the characteristics of the physical deformation in multimodal image registration tasks that are usually continuous and relatively smooth, the computationally intensive convolutional upsampling module is abandoned, and the spatial scale of the low-resolution dense displacement field is magnified to the original image resolution using a bilinear interpolation operator. Then, a two-dimensional average pooling layer is introduced as the final smoother to filter out the local jaggedness and high-frequency noise brought about by the scale magnification process and output a full-resolution smooth displacement field Φ.

[0036] 34) Spatial field composition and effective mask generation: Receive the global affine transformation matrix M output from the first stage and the full-resolution smooth displacement field Φ output from the second stage; generate a global affine sampling grid covering the scale of the original image based on the global affine transformation matrix M, and superimpose the full-resolution smooth displacement field Φ onto the global affine sampling grid. Through spatial grid composition calculation, the predicted joint dense spatial displacement field is obtained.

[0037] Boundary crossing determination is performed on the predicted joint dense spatial displacement field. The coordinate pixels mapped to the physical boundary of the original image are extracted. The internal effective overlapping regions that do not cross the boundary are marked as 1, and the empty pixel background regions that cross the boundary are marked as 0. The predicted registration effective mask is obtained, and the predicted joint dense spatial displacement field and the predicted registration effective mask are output.

[0038] The construction of the hierarchical fusion network architecture based on painting logic is as follows: following a step-by-step reconstruction logic, a hierarchical fusion network is constructed, including a structural encoder, a texture injection module, and a lighting rendering module; within the registered feature space, the geometric skeleton is aggregated, texture details are dynamically injected through spatial feature transformation technology, and the lighting background is fused through a gating mechanism to generate a fused image; it includes the following steps:

[0039] 41) Anti-artifact structural skeleton outlining: Setting S fuse,ir With the registered S fuse,vi The deep structure features are extracted by inputting a dual-branch structure encoder to obtain infrared deep structure features and visible light deep structure features;

[0040] The dual-branch encoder is designed to consist of an initial feature extraction convolutional layer and a cascaded sequence of residual convolutional blocks. Infrared and visible light deep structural features are first aggregated using a maximum value fusion strategy. This strategy leverages the characteristic of maximizing values ​​in non-overlapping regions to naturally suppress the zero-value black background generated by visible light registration. Subsequently, a spatial attention module is introduced. Channel-level max pooling and average pooling are used to extract salient spatial responses and generate a weight map. This weight map is then used to refine the initially aggregated structural features, filtering out background noise and generating a clean high-frequency skeleton feature F.skeleton ;

[0041] 42) Mask pre-filtering and dynamic texture injection:

[0042] The setup uses a dual-branch texture encoder to separately process T ir With the registered T vi Feature extraction is performed to obtain infrared deep texture features and visible light deep texture features;

[0043] The dual-branch texture encoder adopts the same convolutional and residual block cascaded architecture as the dual-branch structure encoder. The predicted registration effective mask is used to perform dot-multiplication filtering on the visible light deep texture features to prevent the step signal at the boundary between the registered black edge and the effective image from being misjudged as strong texture by the convolutional layer. The filtered visible light deep texture features and infrared deep texture features are concatenated along the channel dimension and then mixed using convolution to obtain the joint texture feature F. texture Subsequently, a spatial feature transformation layer is introduced to utilize the high-frequency skeleton feature F. skeleton As a conditional guide, the spatially adaptive scaling factor γ and translation factor β are predicted through a mapping network, using the residual modulation formula F. out = F skeleton + F texture ⊙(1+γ)+β completes the dynamic injection of texture details into skeletal features, where F out The output feature after injecting texture details; ⊙ represents an element-wise multiplication operation.

[0044] 43) Mask-guided lighting rendering and image reconstruction:

[0045] The decoder network is configured to consist of cascaded residual reconstruction blocks, 3×3 output convolutional layers, and hyperbolic tangent activation functions. The texture-injected features are mapped through this decoder network to a high-frequency residual map with values ​​in the range [-1, 1], representing high-frequency details across the entire image. Simultaneously, based on the predicted registration effective mask pair L... ir With the registered L vi Gated reconstruction is performed: the two are mean-fused within the effective overlapping area, while a lighting background fallback mechanism is triggered in the invalid area to fully preserve L. ir The corresponding pixels; finally, the reconstructed low-frequency illumination component and the high-frequency residual map are superimposed at the pixel level and limited to generate the final fused image.

[0046] The training of the registration and fusion model includes the following steps:

[0047] 51) Training a two-stage cascaded registration network based on phased training, specifically including the following steps:

[0048] 511) Training for large-scale affine coarse registration: Training cross-modal images to obtain enhanced structural features S after decoupling extraction. reg,ir and S reg,vi The first stage of the two-stage cascaded registration network architecture is input, namely the iterative global affine regression network based on U-Net. The output global affine transformation matrix is ​​compared with the inverse affine transformation matrix, and the absolute error matrix L1 loss in the parameter space is calculated. At the same time, a standard pixel coordinate grid is established, and the coordinate deviation loss between the target coordinates after the global affine transformation matrix and the inverse affine transformation matrix are calculated. The absolute error matrix L1 loss and the coordinate deviation loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the iterative global affine regression network to complete the training of the iterative global affine regression network and output the predicted global affine transformation matrix.

[0049] 512) Training for Local Elastic Fine Registration: Freeze the weight parameters of the global affine regression network, and set S... reg,ir S after registration with the predicted global affine transformation matrix reg,vi The second stage of the two-stage cascaded registration network architecture is input, namely, the RAFT-based elastic registration network. In each iteration of the elastic registration network, the updated coordinate grid is extracted and the initial absolute coordinate grid is subtracted to obtain the dense displacement field of the current iteration step. The dense displacement fields of all iteration steps are combined to form a multi-step residual displacement field sequence. Combining the registration inverse deformation field corresponding to the artificial deformation and the global affine transformation matrix predicted in the first stage, the corresponding residual displacement field between the two is calculated through spatial grid composite calculation as the true dense displacement field label to be fitted by the elastic registration network in this stage. The L1 distance between the two is calculated under the spatial weighting of the true registration effective mask. The overlapping foreground of the image is given a very high weight while the black border area of ​​the background is ignored. The L1 distance of each iteration step is weighted and summed as the multi-step prediction loss. At the same time, the second gradient of the final displacement field is calculated as the smoothness loss. The multi-step prediction loss and the smoothness loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the elastic registration network to complete the training of the elastic registration network. The predicted joint dense spatial displacement field and the predicted registration effective mask are output.

[0050] 52) Training of hierarchical fusion networks based on teacher-student knowledge distillation specifically includes the following steps:

[0051] 521) Independent pre-training of the teacher hierarchical fusion network: The teacher hierarchical fusion network is set to be trained using the components after decoupling the original registered infrared and visible light images. The original registered infrared and visible light image pairs are decoupled to extract structural, texture and illumination features using the constructed physical heuristic feature decoupling mechanism. The features of each component are input into the initialized teacher hierarchical fusion network. Gradient backpropagation and end-to-end parameter fine-tuning of the teacher hierarchical fusion network are performed on the entire image using a hybrid fusion loss that includes pixel intensity, structural similarity and spatial gradient, thus completing the independent pre-training of the teacher hierarchical fusion network.

[0052] 522) Construction of input features for student hierarchical fusion network: The cross-modal image training pair containing artificial deformation is decoupled using the constructed physical heuristic feature decoupling mechanism. The enhanced structural features of infrared and visible light are input into the pre-trained iterative global affine regression network and elastic registration network to obtain the predicted joint dense spatial displacement field and registration effective mask. The features of each component of visible light are resampled and registered using the predicted joint dense spatial displacement field. The registered features of each component of visible light and the features of each component of infrared light are jointly input into the student hierarchical fusion network to be optimized to output the fused image.

[0053] 523) Generation of high-quality soft labels: In each training iteration, the original registered infrared and visible light image pairs are decoupled from the structure, texture and illumination features using the constructed physical heuristic feature decoupling mechanism. Each feature is then input into the frozen teacher layered fusion network. The high-quality fusion image output by the teacher layered fusion network, which is not affected by registration distortion and boundary artifacts, is used as a soft label.

[0054] 524) Distillation Loss Calculation and Parameter Fine-tuning Based on Mask Intersection: Obtain the fused image output by the student hierarchical fusion network, and perform a logical AND operation between the real registration effective mask and the predicted registration effective mask to generate a mask intersection; within the area marked by the mask intersection, calculate the L1 distillation loss between the fused image output by the student hierarchical fusion network and the soft label output by the teacher hierarchical fusion network; combine the distillation loss with the hybrid fusion loss within the mask intersection area, and perform gradient backpropagation and end-to-end parameter fine-tuning on the student hierarchical fusion network. The trained student hierarchical fusion network is the hierarchical fusion network based on drawing logic.

[0055] A computer-readable storage medium storing a computer program that, when executed by a processor, enables a method for infrared and visible light image registration and fusion based on physically heuristic feature decoupling.

[0056] A computer device is characterized by comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it can realize a method for infrared and visible light image registration and fusion based on physical heuristic feature decoupling.

[0057] Beneficial effects

[0058] This invention presents a physical heuristic feature decoupling-based infrared and visible light image registration and fusion method, which creatively establishes a complete technical system of "physically driven decoupling - cascaded recursive registration - painterly reconstruction fusion" compared with existing technologies. First, it utilizes imaging physics to decompose complex cross-modal images into three orthogonal components: illumination, structure, and texture, eliminating the interference of modal differences on registration at the source. Second, it constructs a cascaded registration architecture consisting of a global affine transformation network and a recursive transformation network based on the MIND-SSC descriptor, achieving a leap from global large-scale displacement correction to local pixel-level elastic alignment. Finally, it proposes a layered fusion network that simulates human painting logic, dynamically injecting texture details into the geometric skeleton through spatial feature transformation technology, ultimately achieving high-fidelity, ghosting-free fusion in large-scale mismatch scenes.

[0059] The present invention also has the following advantages:

[0060] 1. Strong physical interpretability and robust feature extraction: This invention abandons the traditional "black box" feature extraction method and explicitly separates the illumination, structure, and texture components through a physically heuristic decoupling mechanism. In particular, the structural features generated by L0 smoothing and CLAHE enhancement effectively filter out the interference of infrared thermal diffusion noise and complex visible light background, bridging the modal gap at its source and providing a unified benchmark with geometric invariance for registration.

[0061] 2. High robustness to large-scale misalignment and precise alignment: Addressing the limitations of existing methods in handling large displacements, the cascaded registration architecture designed in this invention breaks through traditional limitations. Global affine coarse registration can quickly correct large-scale rotation and translation deviations, pulling the image back to the nearest neighbor region; combined with elastic fine registration based on MIND-SSC descriptors and GRU recursive iteration, it can accurately capture minute nonlinear deformations, achieving pixel-level precise alignment from global to local.

[0062] 3. High fidelity fusion and effective artifact elimination: The painterly layered fusion strategy proposed in this invention changes the simple pixel overlay paradigm. Through Spatial Feature Transform (SFT) technology, texture details are dynamically injected with geometric skeletons as constraints. Combined with mask-based lighting rendering, ghosting and edge blurring caused by small registration residuals can be effectively eliminated, while avoiding black border problems caused by resampling. This ensures that the fused image has both significant infrared targets and clear visible light textures. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the physical heuristic feature decoupling mechanism of the present invention;

[0064] Figure 2 This is a diagram of the two-stage cascaded registration network architecture of the present invention;

[0065] Figure 3 This is a diagram of the hierarchical fusion network architecture of the present invention.

[0066] Figure 4 This is a sequence diagram of the method of the present invention. Detailed Implementation

[0067] To provide a better understanding of the structural features and effects achieved by the present invention, a detailed description is provided below, accompanied by preferred embodiments and accompanying drawings:

[0068] like Figure 4 As shown, the infrared and visible light image registration and fusion method based on physically heuristic feature decoupling described in this invention includes the following steps:

[0069] The first step is the acquisition and preprocessing of infrared and visible light images: the registered infrared and visible light image source data are acquired and adjusted to a specified size. The infrared image is kept as a reference. The visible light image is artificially subjected to random global affine transformation and local elastic deformation through spatial geometric transformation technology to generate cross-modal image training pairs containing unregistered images. At the same time, the inverse affine transformation matrix corresponding to the artificial deformation, the inverse deformation field of registration, and the true registration effective mask of the effective area of ​​the registered image are calculated and generated.

[0070] The second step is to construct a physically heuristic feature decoupling mechanism for cross-modal image processing.

[0071] Existing technologies for cross-modal image processing often treat the entire image as input, ignoring the physical differences in the representation of thermal radiation in infrared images and the representation of reflected light and rich textures in visible light images. This "modal gap" easily leads to feature extraction failures or mismatches. To address this, this invention innovatively designs a physically heuristic feature decoupling mechanism, which greatly reduces the difficulty of subsequent network processing of heterogeneous data and provides a geometrically invariant unified benchmark for high-precision registration.

[0072] like Figure 1As shown, based on the imaging physics mechanism, the input cross-modal image training pair is explicitly decomposed into illumination feature components carrying illumination information, structural feature components carrying geometric contours, and texture feature components carrying modality-specific details. Simultaneously, the decoupling process also generates enhanced structural features, collectively providing a unified feature benchmark for cross-modal registration and fusion. The specific steps are as follows:

[0073] (1) Convolve the cross-modal image training pair using a large-scale Gaussian low-pass filter to extract the low-frequency illumination component L, and subtract the low-frequency illumination component L from the original image to obtain the high-frequency detail component H.

[0074] (2) The L0 norm smoothing operator is introduced to optimize the high-frequency detail component H. By minimizing the non-zero pixel statistics of the gradient, heat diffusion noise and low contrast texture are filtered out. Combined with the contrast-limited adaptive histogram equalization technique, the enhanced structural feature component S is generated. reg ;

[0075] (3) With enhanced structural features S reg To guide the image, guided filtering techniques are used to decompose the high-frequency detail component H to obtain structural features S. fuse and texture feature T;

[0076] (4) After decoupling, the cross-modal image training pairs yielded eight feature components, as follows: Visible light image enhanced structural features S reg,vi Infrared image enhanced structural features S reg,ir Visible light image structural features S fuse,vi Infrared image structural features S fuse,ir Visible light image texture features T vi Infrared image texture features T ir Visible light image illumination features L vi Infrared image illumination features L ir .

[0077] The third step is to construct a two-stage cascaded registration network architecture based on decoupling features.

[0078] Image distortion in real-world scenarios is often a combination of large-scale viewpoint shifts (such as translation and rotation) and local elastic distortions (such as lens distortion). A single network cannot simultaneously achieve both a large receptive field and high-resolution accuracy.

[0079] To address this, the present invention proposes a two-stage cascaded registration network architecture based on decoupling features, such as... Figure 2As shown, a cascaded deep learning architecture is adopted. The first stage utilizes the decoupled enhanced structural features to predict the affine matrix through an iterative global affine regression network based on U-Net, correcting large-scale linear offsets between images. The second stage introduces a RAFT-based elastic registration network to iteratively refine the displacement field in a single-scale feature space, capturing and correcting local nonlinear elastic deformations. The two stages complement each other, effectively solving the registration problem under large-scale complex distortions. The specific steps are as follows:

[0080] (1) The two-stage cascaded registration network architecture is set up, including the first stage of the U-Net-based iterative global affine regression network and the second stage of the RAFT-based elastic registration network.

[0081] (2) Setting up an iterative global affine regression network based on U-Net:

[0082] A1) Initial State Setting and Dynamic Joint Feature Construction: Set the input of the affine regression network to S reg,vi and S reg,ir The initial cumulative affine transformation matrix is ​​the identity matrix; in each iteration, the cumulative affine transformation matrix from the previous iteration step is first applied to S. reg,vi A spatial transformation is performed to obtain the visible light features in the current aligned state; subsequently, S is extracted respectively. reg,ir The Sobel spatial gradient of the visible light features in the current aligned state is calculated and averaged to generate a structural gradient guidance map; finally, S... reg,ir The visible light features in the current alignment state and the structural gradient guidance Figure 3 The components are concatenated along the channel dimension and used as the joint input to the iterative global affine regression network of the current iteration step.

[0083] A2) Residual parameter regression based on U-Net bottleneck layer: Modify lightweight U-Net as feature extractor, discard the image reconstruction branch of decoder, feed the joint input into encoder for downsampling, extract bottleneck layer features of deep global geometric context, and generate two-dimensional bottleneck layer feature map; then, use adaptive global average pooling layer to compress the two-dimensional bottleneck layer feature map into one-dimensional feature vector, and predict affine transformation residual increment parameters through linear regression head initialized to zero weights.

[0084] A3) Homogeneous Matrix Chain Multiplication and Iterative Correction: In each iteration, the predicted affine transformation residual increment parameters are analyzed as rotation scaling increments and translation increments, and constructed as the increment affine matrix for the current iteration step; the increment affine matrix and the cumulative affine transformation matrix of the previous iteration step are both converted into 3×3 homogeneous transformation matrices, and composite updates are completed through homogeneous matrix multiplication to obtain the cumulative affine transformation matrix for the current iteration step; the latest cumulative matrix is ​​output to the next iteration for visible light feature resampling in step A1); through a set number of multi-step recursive iterations, the true deformation is gradually approximated, and finally the global affine transformation matrix M that eliminates large-scale linear offset is output;

[0085] (3) Set up a RAFT-based flexible registration network:

[0086] B1) Modality-independent feature extraction: Set the input of the elastic registration network to S reg,ir And S after affine transformation matrix M reg,vi ;

[0087] The flexible registration network is configured to include a non-shared feature encoder, a MIND-SSC module, and a shared feature encoder.

[0088] The non-shared feature encoder comprises downsampling convolutional layers with large-scale convolutional kernels, instance normalization layers, and residual convolutional blocks, used to process the input S... reg,ir And S after global affine transformation matrix M reg,vi Initial spatial dimensionality reduction and independent feature mapping are performed to obtain single-channel features for both modalities;

[0089] The single-channel features of the two modalities are input into the MIND-SSC module, and the structural self-similarity distance in the local neighborhood of the pixel is calculated based on multiple preset spatial offset directions to generate a multi-channel structural descriptor.

[0090] The shared feature encoder includes cascaded downsampled residual convolutional blocks for further spatial dimensionality reduction and deep common feature extraction of the structural descriptor. Two 1×1 convolutional layers are connected in parallel at the end of the shared feature encoder to output infrared matching features and visible light matching features for calculating similarity, as well as contextual features for guiding the update of the hidden layer state.

[0091] B2) Coordinate Grid Initialization and Dynamic Local Cost Volume Construction: Before the start of the loop iteration, a standard two-dimensional pixel grid with the same spatial dimension as the input feature is generated as the initial absolute coordinate grid; the infrared matching feature and visible light matching feature output from step B1) are received and pre-smoothed by two-dimensional Gaussian filtering to suppress multimodal feature-level noise; in each iteration, a local search grid is generated within the set local search radius with the absolute coordinate grid of the current iteration step as the sampling center; the visible light matching feature is spatially resampled using this local search grid, and the L2 squared distance between the resampled feature and the infrared matching feature in the channel dimension is calculated to dynamically construct a single-scale local cost volume, which is then output as an error signal;

[0092] B3) ConvGRU iterative loop and displacement field update:

[0093] An update block is defined, which includes a feature fusion convolutional layer, a gated recurrent unit, and a flow field regression head. In each iteration, the absolute coordinate grid of the current iteration step is first subtracted from the initial absolute coordinate grid to calculate the dense displacement field of the current iteration step. Then, the feature fusion convolutional layer is used to perform convolutional dimensionality reduction on the local cost convolution in step B2) and the dense displacement field of the current iteration step, and concatenated with the context features output in step B1) in the channel dimension as the joint input of ConvGRU. ConvGRU uses the internal convolutional update gate and reset gate to infer the error gradient in combination with the hidden state of the previous round and outputs the updated hidden state. The hidden state is fed into the flow field regression head containing multiple convolutions to output the residual update amount of the dense displacement field. The residual update amount is superimposed on the absolute coordinate grid of the current iteration step to obtain the updated absolute coordinate grid, and it is returned to step B2) as the sampling center of the next iteration, forming a closed loop, gradually approximating the real nonlinear elastic deformation.

[0094] B4) Displacement field upsampling and smoothing mechanism: After the loop iteration is completed, the updated coordinate grid of the last output in step B3) is subtracted from the initial absolute coordinate grid to obtain a low-resolution dense displacement field. In order to adapt to the characteristics of the physical deformation in multimodal image registration tasks that are usually continuous and relatively smooth, the computationally intensive convolutional upsampling module is abandoned, and the spatial scale of the low-resolution dense displacement field is magnified to the original image resolution using a bilinear interpolation operator. Then, a two-dimensional average pooling layer is introduced as the final smoother to filter out the local jaggedness and high-frequency noise brought about by the scale magnification process and output a full-resolution smooth displacement field Φ.

[0095] (4) Spatial field composition and effective mask generation: Receive the global affine transformation matrix M output from the first stage and the full-resolution smooth displacement field Φ output from the second stage; generate a global affine sampling grid covering the scale of the original image based on the global affine transformation matrix M, and superimpose the full-resolution smooth displacement field Φ onto the global affine sampling grid. Through spatial grid composition calculation, the predicted joint dense spatial displacement field is obtained.

[0096] Boundary crossing determination is performed on the predicted joint dense spatial displacement field. The coordinate pixels mapped to the physical boundary of the original image are extracted. The internal effective overlapping regions that do not cross the boundary are marked as 1, and the empty pixel background regions that cross the boundary are marked as 0. The predicted registration effective mask is obtained, and the predicted joint dense spatial displacement field and the predicted registration effective mask are output.

[0097] The fourth step is to construct a layered fusion network architecture based on painting logic.

[0098] Existing infrared and visible light image registration and fusion methods typically feed the registered images directly into a fusion network for fusion. This brute-force mixing at the image level can easily lead to gradient conflicts between prominent infrared targets and visible light texture details, resulting in decreased contrast of thermal targets or smearing of high-frequency textures. At the same time, large-scale registration can produce black-bordered background pixels with zero-value filling, which can spread under the convolutional receptive field to form severe artifacts and ghosting.

[0099] To address the aforementioned issues, this invention fully utilizes the component features of the infrared and visible light images decoupled through the aforementioned physical heuristic feature decoupling mechanism. It simulates the logic of "first outlining the skeleton, then adding texture, and finally rendering the lighting and shadows" in painting, implementing differentiated customized fusion strategies for components with different physical properties. Simultaneously, a gating mechanism based on effective registration masks is introduced to achieve hard backtracking of the illumination background. This step-by-step reconstruction and mask filtering effectively blocks the propagation of distortion residuals to the fusion result, ensuring the high fidelity of the final image.

[0100] like Figure 3 As shown, following a step-by-step reconstruction logic, a hierarchical fusion network is constructed, comprising a structural encoder, a texture injection module, and a lighting rendering module. Within the registered feature space, the geometric skeleton is aggregated, texture details are dynamically injected using spatial feature transformation techniques, and the lighting background is fused through a gating mechanism to generate a fused image. The specific steps are as follows:

[0101] (1) Anti-artifact structural skeleton delineation: Set S fuse,ir With the registered S fuse,vi The deep structure features are extracted by inputting a dual-branch structure encoder to obtain infrared deep structure features and visible light deep structure features;

[0102] The dual-branch encoder is designed to consist of an initial feature extraction convolutional layer and a cascaded sequence of residual convolutional blocks. Infrared and visible light deep structural features are first aggregated using a maximum value fusion strategy. This strategy leverages the characteristic of maximizing values ​​in non-overlapping regions to naturally suppress the zero-value black background generated by visible light registration. Subsequently, a spatial attention module is introduced. Channel-level max pooling and average pooling are used to extract salient spatial responses and generate a weight map. This weight map is then used to refine the initially aggregated structural features, filtering out background noise and generating a clean high-frequency skeleton feature F. skeleton ;

[0103] (2) Mask pre-filtering and dynamic texture injection:

[0104] The setup uses a dual-branch texture encoder to separately process T ir With the registered T vi Feature extraction is performed to obtain infrared deep texture features and visible light deep texture features;

[0105] The dual-branch texture encoder adopts the same convolutional and residual block cascaded architecture as the dual-branch structure encoder. The predicted registration effective mask is used to perform dot-multiplication filtering on the visible light deep texture features to prevent the step signal at the boundary between the registered black edge and the effective image from being misjudged as strong texture by the convolutional layer. The filtered visible light deep texture features and infrared deep texture features are concatenated along the channel dimension and then mixed using convolution to obtain the joint texture feature F. texture Subsequently, a spatial feature transformation layer is introduced to utilize the high-frequency skeleton feature F. skeleton As a conditional guide, the spatially adaptive scaling factor γ and translation factor β are predicted through a mapping network, using the residual modulation formula F. out = F skeleton + F texture ⊙(1+γ)+β completes the dynamic injection of texture details into skeletal features, where F out The output feature after injecting texture details; ⊙ represents an element-wise multiplication operation.

[0106] (3) Mask-guided lighting rendering and image reconstruction:

[0107] The decoder network is configured to consist of cascaded residual reconstruction blocks, 3×3 output convolutional layers, and hyperbolic tangent activation functions. The texture-injected features are mapped through this decoder network to a high-frequency residual map with values ​​in the range [-1, 1], representing high-frequency details across the entire image. Simultaneously, based on the predicted registration effective mask pair L... ir With the registered L vi Gated reconstruction is performed: the two are mean-fused within the effective overlapping area, while a lighting background fallback mechanism is triggered in the invalid area to fully preserve L. irThe corresponding pixels; finally, the reconstructed low-frequency illumination component and the high-frequency residual map are superimposed at the pixel level and limited to generate the final fused image.

[0108] The fifth step is training the registration and fusion model.

[0109] Deep learning models, facing highly coupled multi-task scenarios like registration and fusion, are prone to gradient chaos and local optima if directly trained end-to-end. To address this, this invention employs a phased pre-training and teacher-student knowledge distillation strategy. First, by independently optimizing the iterative global affine regression network and the elastic registration network sequentially, robust spatial alignment of cross-modal features under complex deformations is ensured. Second, to overcome the black border artifacts caused by registration, a teacher-student distillation strategy is innovatively introduced. An ideal image without registration error guides the teacher network to generate a perfect fusion prior, and the intersection of the real masks is used as a loss constraint to guide the student network processing distorted images to perform local imitation learning, enabling it to learn perfect fusion rendering logic even under complex geometric distortions. The specific steps are as follows:

[0110] (1) Training of a two-stage cascaded registration network based on phased training, specifically including the following steps:

[0111] C1) Training for large-scale affine coarse registration: Training cross-modal images onto the enhanced structural features S obtained after decoupling extraction. reg,ir and S reg,vi The first stage of the two-stage cascaded registration network architecture is input, namely the iterative global affine regression network based on U-Net. The output global affine transformation matrix is ​​compared with the inverse affine transformation matrix, and the absolute error matrix L1 loss in the parameter space is calculated. At the same time, a standard pixel coordinate grid is established, and the coordinate deviation loss between the target coordinates after the global affine transformation matrix and the inverse affine transformation matrix are calculated. The absolute error matrix L1 loss and the coordinate deviation loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the iterative global affine regression network to complete the training of the iterative global affine regression network and output the predicted global affine transformation matrix.

[0112] C2) Training for Local Elastic Fine Registration: Freeze the weight parameters of the global affine regression network, and set S... reg,ir S after registration with the predicted global affine transformation matrix reg,viThe second stage of the two-stage cascaded registration network architecture is input, namely, the RAFT-based elastic registration network. In each iteration of the elastic registration network, the updated coordinate grid is extracted and the initial absolute coordinate grid is subtracted to obtain the dense displacement field of the current iteration step. The dense displacement fields of all iteration steps are combined to form a multi-step residual displacement field sequence. Combining the registration inverse deformation field corresponding to the artificial deformation and the global affine transformation matrix predicted in the first stage, the corresponding residual displacement field between the two is calculated through spatial grid composite calculation as the true dense displacement field label to be fitted by the elastic registration network in this stage. The L1 distance between the two is calculated under the spatial weighting of the true registration effective mask. The overlapping foreground of the image is given a very high weight while the black border area of ​​the background is ignored. The L1 distance of each iteration step is weighted and summed as the multi-step prediction loss. At the same time, the second gradient of the final displacement field is calculated as the smoothness loss. The multi-step prediction loss and the smoothness loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the elastic registration network to complete the training of the elastic registration network. The predicted joint dense spatial displacement field and the predicted registration effective mask are output.

[0113] (2) Training of hierarchical fusion networks based on teacher-student knowledge distillation, specifically including the following steps:

[0114] D1) Independent pre-training of the teacher hierarchical fusion network: The teacher hierarchical fusion network is trained using the components of the original registered infrared and visible light images after decoupling. The original registered infrared and visible light image pairs are decoupled to extract structural, texture, and lighting features using the constructed physical heuristic feature decoupling mechanism. The features of each component are input into the initialized teacher hierarchical fusion network. Gradient backpropagation and end-to-end parameter fine-tuning of the teacher hierarchical fusion network are performed on the entire image using a hybrid fusion loss that includes pixel intensity, structural similarity, and spatial gradient, thus completing the independent pre-training of the teacher hierarchical fusion network.

[0115] D2) Construction of input features for the student hierarchical fusion network: The cross-modal image training pair containing artificial deformation is decoupled using the constructed physical heuristic feature decoupling mechanism. The enhanced structural features of infrared and visible light are input into the pre-trained iterative global affine regression network and elastic registration network to obtain the predicted joint dense spatial displacement field and registration effective mask. The features of each component of visible light are resampled and registered using the predicted joint dense spatial displacement field. The registered features of each component of visible light and each component of infrared light are then jointly input into the student hierarchical fusion network to be optimized to output the fused image.

[0116] D3) Generation of high-quality soft labels: In each training iteration, the original registered infrared and visible light image pairs are decoupled from the structure, texture and illumination features using the constructed physical heuristic feature decoupling mechanism. Each feature is then input into the frozen teacher layered fusion network. The high-quality fused image output by the teacher layered fusion network, which is not affected by registration distortion and boundary artifacts, is used as a soft label.

[0117] D4) Distillation Loss Calculation and Parameter Fine-Tuning Based on Mask Intersection: Obtain the fused image output by the student hierarchical fusion network, and perform a logical AND operation between the real registration effective mask and the predicted registration effective mask to generate a mask intersection; within the area marked by the mask intersection, calculate the L1 distillation loss between the fused image output by the student hierarchical fusion network and the soft label output by the teacher hierarchical fusion network; combine this distillation loss with the hybrid fusion loss within the mask intersection area, and perform gradient backpropagation and end-to-end parameter fine-tuning on the student hierarchical fusion network. The trained student hierarchical fusion network is the hierarchical fusion network based on drawing logic.

[0118] Step 6, Acquisition and preprocessing of the images to be registered and fused: Acquire the unregistered infrared and visible light images to be processed in the actual scene, and perform physical heuristic feature decoupling using a physical heuristic feature decoupling mechanism.

[0119] Step 7: Obtaining the registration and fusion image results: The preprocessed structural features to be registered are input into a two-stage cascaded registration network, which outputs the predicted joint dense spatial displacement field and the effective registration mask. The joint dense spatial displacement field is applied to the visible light component features to be registered to achieve registration. Finally, the registered visible light component features, infrared component features, and the effective mask are input into a hierarchical fusion network. Feature-level reconstruction is performed within the effective area indicated by the mask, and the final high-quality registered and fused image is output.

[0120] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for infrared and visible light image registration and fusion based on physically heuristic feature decoupling, characterized in that, Includes the following steps: 11) Acquisition and preprocessing of infrared and visible light images: Acquire registered infrared and visible light image source data and adjust them to a specified size. Keep the infrared image as a reference. Apply random global affine transformation and local elastic deformation to the visible light image through spatial geometric transformation technology to generate cross-modal image training pairs containing unregistered images. At the same time, calculate and generate the inverse affine transformation matrix corresponding to the artificial deformation, the inverse deformation field of registration, and the true registration effective mask of the effective area of ​​the registered image. 12) Construct a physically heuristic feature decoupling mechanism for cross-modal image processing; 13) Construct a two-stage cascaded registration network architecture based on decoupling features; 14) Construct a hierarchical fusion network architecture based on painting logic; 15) Training of the registration and fusion model: A phased training and teacher-student knowledge distillation strategy is adopted for training. First, the iterative global affine regression network is trained for large-scale affine coarse registration. Then, the iterative global affine regression network is frozen, and the elastic registration network is trained for local elastic fine registration. Finally, the iterative global affine regression network and the elastic registration network are frozen, and the registered source images are used as teacher labels to train the hierarchical fusion network for teacher-student knowledge distillation. 16) Acquisition and preprocessing of the image to be registered and fused: Acquire the unregistered infrared and visible light images to be processed in the actual scene, and use the mechanism of step 12) to perform physical heuristic feature decoupling; 17) Obtaining the registration and fusion image results: The preprocessed structural features to be registered are input into a two-stage cascaded registration network, which outputs the predicted joint dense spatial displacement field and the effective registration mask. The joint dense spatial displacement field is applied to the visible light component features to be registered to achieve registration. Finally, the registered visible light component features, infrared component features and the effective mask are input into a hierarchical fusion network. Feature-level reconstruction is performed in the effective area indicated by the mask, and the final high-quality registration and fusion image is output.

2. The infrared and visible light image registration and fusion method based on physically heuristic feature decoupling according to claim 1, characterized in that, The physical heuristic feature decoupling mechanism for cross-modal image processing is constructed as follows: based on the imaging physics mechanism, the input cross-modal image training pair is explicitly decomposed into illumination feature components carrying illumination information, structural feature components carrying geometric contours, and texture feature components carrying modality-specific details. At the same time, the decoupling process also generates enhanced structural features, which together provide a unified feature benchmark for cross-modal registration and fusion. It includes the following steps: 21) Convolve the cross-modal image training pair using a large-scale Gaussian low-pass filter to extract the low-frequency illumination component L, and subtract the low-frequency illumination component L from the original image to obtain the high-frequency detail component H. 22) An L0 norm smoothing operator is introduced to optimize the high-frequency detail component H. By minimizing the non-zero pixel statistics of the gradient, thermal diffusion noise and low-contrast textures are filtered out. Combined with contrast-limited adaptive histogram equalization, an enhanced structural feature component S is generated. reg ; 23) With enhanced structural features S reg To guide the image, guided filtering techniques are used to decompose the high-frequency detail component H to obtain structural features S. fuse and texture feature T; 24) After decoupling, the cross-modal image training pairs yielded eight feature components, as follows: Visible light image enhanced structural features S reg,vi Infrared image enhanced structural features S reg,ir Visible light image structural features S fuse,vi Infrared image structural features S fuse,ir Visible light image texture features T vi Infrared image texture features T ir Visible light image illumination features L vi Infrared image illumination features L ir .

3. The infrared and visible light image registration and fusion method based on physically heuristic feature decoupling according to claim 2, characterized in that, The two-stage cascaded registration network architecture based on decoupling features is constructed as follows: a cascaded deep learning architecture is adopted, and the enhanced structural features after decoupling are used to predict the affine matrix through an iterative global affine regression network based on U-Net to correct the large-scale linear offset between images; an elastic registration network based on RAFT is introduced to iteratively refine the displacement field in a single-scale feature space to capture and correct local nonlinear elastic deformation. It includes the following steps: 31) The two-stage cascaded registration network architecture is set up, including the first stage of an iterative global affine regression network based on U-Net and the second stage of an elastic registration network based on RAFT. 32) Setting up an iterative global affine regression network based on U-Net: 321) Initial State Setting and Dynamic Joint Feature Construction: Set the input of the affine regression network to S reg,vi and S reg,ir The initial cumulative affine transformation matrix is ​​the identity matrix; in each iteration, the cumulative affine transformation matrix from the previous iteration step is first applied to S. reg,vi A spatial transformation is performed to obtain the visible light features in the current aligned state; subsequently, S is extracted respectively. reg,ir The Sobel spatial gradient of the visible light features in the current aligned state is calculated and averaged to generate a structural gradient guidance map; finally, S... reg,ir The visible light features in the current alignment state and the structural gradient guidance map are concatenated in the channel dimension and used as the joint input of the iterative global affine regression network in the current iteration step. 322) Residual parameter regression based on U-Net bottleneck layer: Modify the lightweight U-Net as the feature extractor, discard the image reconstruction branch of the decoder, feed the joint input into the encoder for downsampling, extract the bottleneck layer features of the deep global geometric context, and generate a two-dimensional bottleneck layer feature map; then, use an adaptive global average pooling layer to compress the two-dimensional bottleneck layer feature map into a one-dimensional feature vector, and predict the affine transformation residual increment parameters through a linear regression head initialized to zero weights; 323) Homogeneous Matrix Chain Multiplication and Iterative Correction: In each iteration, the predicted affine transformation residual increment parameters are analyzed as rotation scaling increments and translation increments, and constructed as the increment affine matrix for the current iteration step; the increment affine matrix and the cumulative affine transformation matrix of the previous iteration step are both converted into 3×3 homogeneous transformation matrices, and composite updates are completed through homogeneous matrix multiplication to obtain the cumulative affine transformation matrix for the current iteration step; the latest cumulative matrix is ​​output to the next iteration for visible light feature resampling in step 321); through a set number of multi-step recursive iterations, the true deformation is gradually approximated, and finally the global affine transformation matrix M that eliminates large-scale linear offset is output; 33) Setting up a RAFT-based flexible registration network: 331) Modality-independent feature extraction: Set the input of the elastic registration network to S reg,ir And S after affine transformation matrix M reg,vi ; The flexible registration network is configured to include a non-shared feature encoder, a MIND-SSC module, and a shared feature encoder. The non-shared feature encoder comprises downsampling convolutional layers with large-scale convolutional kernels, instance normalization layers, and residual convolutional blocks, used to process the input S... reg,ir And S after global affine transformation matrix M reg,vi Initial spatial dimensionality reduction and independent feature mapping are performed to obtain single-channel features for both modalities; The single-channel features of the two modalities are input into the MIND-SSC module, and the structural self-similarity distance in the local neighborhood of the pixel is calculated based on multiple preset spatial offset directions to generate a multi-channel structural descriptor. The shared feature encoder includes cascaded downsampled residual convolutional blocks for further spatial dimensionality reduction and deep common feature extraction of the structural descriptor. Two 1×1 convolutional layers are connected in parallel at the end of the shared feature encoder to output infrared matching features and visible light matching features for calculating similarity, as well as contextual features for guiding the update of the hidden layer state. 332) Initialization of coordinate grid and dynamic local cost volume: Before the start of the loop iteration, a standard two-dimensional pixel grid with the same dimension as the input feature space is generated as the initial absolute coordinate grid; the infrared matching feature and visible light matching feature output from step 331) are set to be received and pre-smoothed by two-dimensional Gaussian filtering to suppress multimodal feature-level noise; in each iteration, a local search grid is generated within the set local search radius with the absolute coordinate grid of the current iteration step as the sampling center; the visible light matching feature is spatially resampled using the local search grid, and the L2 squared distance between the resampled feature and the infrared matching feature in the channel dimension is calculated to dynamically construct a single-scale local cost volume, and the local cost volume is output as an error signal; 333) ConvGRU iterative loop and displacement field update: An update block is defined, which includes a feature fusion convolutional layer, a gated recurrent unit, and a flow field regression head. In each iteration, the absolute coordinate grid of the current iteration step is first subtracted from the initial absolute coordinate grid to calculate the dense displacement field of the current iteration step. Then, the feature fusion convolutional layer is used to perform convolutional dimensionality reduction on the local cost convolution in step 332) and the dense displacement field of the current iteration step, and concatenated with the context features output in step 331) in the channel dimension as the joint input of ConvGRU. ConvGRU uses the internal convolutional update gate and reset gate to infer the error gradient in combination with the hidden state of the previous round and outputs the updated hidden state. The hidden state is fed into the flow field regression head containing multiple convolutions to output the residual update amount of the dense displacement field. The residual update amount is superimposed on the absolute coordinate grid of the current iteration step to obtain the updated absolute coordinate grid, and it is returned to step 332) as the sampling center of the next iteration, forming a closed loop, gradually approximating the real nonlinear elastic deformation. 334) Displacement field upsampling and smoothing mechanism: After the loop iteration is completed, the updated coordinate grid of the last output in step 333) is subtracted from the initial absolute coordinate grid to obtain a low-resolution dense displacement field. In order to adapt to the characteristics of the physical deformation in multimodal image registration tasks that are usually continuous and relatively smooth, the computationally intensive convolutional upsampling module is abandoned, and the spatial scale of the low-resolution dense displacement field is magnified to the original image resolution using a bilinear interpolation operator. Then, a two-dimensional average pooling layer is introduced as the final smoother to filter out the local jaggedness and high-frequency noise brought about by the scale magnification process and output a full-resolution smooth displacement field Φ. 34) Spatial field composition and effective mask generation: Receive the global affine transformation matrix M output from the first stage and the full-resolution smooth displacement field Φ output from the second stage; generate a global affine sampling grid covering the scale of the original image based on the global affine transformation matrix M, and superimpose the full-resolution smooth displacement field Φ onto the global affine sampling grid. Through spatial grid composition calculation, the predicted joint dense spatial displacement field is obtained. Boundary crossing determination is performed on the predicted joint dense spatial displacement field. The coordinate pixels mapped to the physical boundary of the original image are extracted. The internal effective overlapping regions that do not cross the boundary are marked as 1, and the empty pixel background regions that cross the boundary are marked as 0. The predicted registration effective mask is obtained, and the predicted joint dense spatial displacement field and the predicted registration effective mask are output.

4. The infrared and visible light image registration and fusion method based on physically heuristic feature decoupling according to claim 3, characterized in that, The construction of the hierarchical fusion network architecture based on painting logic is as follows: following the step-by-step reconstruction logic, a hierarchical fusion network is constructed, which includes a structural encoder, a texture injection module, and a lighting rendering module; within the registered feature space, the geometric skeleton is aggregated, texture details are dynamically injected through spatial feature transformation technology, and the lighting background is fused through a gating mechanism to generate a fused image; It includes the following steps: 41) Anti-artifact structural skeleton outlining: Setting S fuse,ir With the registered S fuse,vi The deep structure features are extracted by inputting a dual-branch structure encoder to obtain infrared deep structure features and visible light deep structure features; The dual-branch encoder is designed to consist of an initial feature extraction convolutional layer and a cascaded sequence of residual convolutional blocks. Infrared and visible light deep structural features are first aggregated using a maximum value fusion strategy. This strategy leverages the characteristic of maximizing values ​​in non-overlapping regions to naturally suppress the zero-value black background generated by visible light registration. Subsequently, a spatial attention module is introduced. Channel-level max pooling and average pooling are used to extract salient spatial responses and generate a weight map. This weight map is then used to refine the initially aggregated structural features, filtering out background noise and generating a clean high-frequency skeleton feature F. skeleton ; 42) Mask pre-filtering and dynamic texture injection: The setup uses a dual-branch texture encoder to separately process T... ir With the registered T vi Feature extraction is performed to obtain infrared deep texture features and visible light deep texture features; The dual-branch texture encoder adopts the same convolutional and residual block cascaded architecture as the dual-branch structure encoder. The predicted registration effective mask is used to perform dot-multiplication filtering on the visible light deep texture features to prevent the step signal at the boundary between the registered black edge and the effective image from being misjudged as strong texture by the convolutional layer. The filtered visible light deep texture features and infrared deep texture features are concatenated along the channel dimension and then mixed using convolution to obtain the joint texture feature F. texture Subsequently, a spatial feature transformation layer is introduced to utilize the high-frequency skeleton feature F. skeleton As a conditional guide, the spatially adaptive scaling factor γ and translation factor β are predicted through a mapping network, using the residual modulation formula F. out = F skeleton + F texture ⊙(1+γ)+β completes the dynamic injection of texture details into skeletal features, where F out The output feature after injecting texture details; ⊙ represents an element-wise multiplication operation. 43) Mask-guided lighting rendering and image reconstruction: The decoder network is configured to consist of cascaded residual reconstruction blocks, 3×3 output convolutional layers, and hyperbolic tangent activation functions. The texture-injected features are mapped through this decoder network to a high-frequency residual map with values ​​in the range [-1, 1], representing high-frequency details across the entire image. Simultaneously, based on the predicted registration effective mask pair L... ir With the registered L vi Gated reconstruction is performed: the two are mean-fused within the effective overlapping area, while a lighting background fallback mechanism is triggered in the invalid area to fully preserve L. ir The corresponding pixels; finally, the reconstructed low-frequency illumination component and the high-frequency residual map are superimposed at the pixel level and limited to generate the final fused image.

5. The infrared and visible light image registration and fusion method based on physically heuristic feature decoupling according to claim 1, characterized in that, The training of the registration and fusion model includes the following steps: 51) Training a two-stage cascaded registration network based on phased training, specifically including the following steps: 511) Training for large-scale affine coarse registration: Training cross-modal images to obtain enhanced structural features S after decoupling extraction. reg,ir and S reg,vi The first stage of the two-stage cascaded registration network architecture is input, namely the iterative global affine regression network based on U-Net. The output global affine transformation matrix is ​​compared with the inverse affine transformation matrix, and the absolute error matrix L1 loss in the parameter space is calculated. At the same time, a standard pixel coordinate grid is established, and the coordinate deviation loss between the target coordinates after the global affine transformation matrix and the inverse affine transformation matrix are calculated. The absolute error matrix L1 loss and the coordinate deviation loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the iterative global affine regression network to complete the training of the iterative global affine regression network and output the predicted global affine transformation matrix. 512) Training for Local Elastic Fine Registration: Freeze the weight parameters of the global affine regression network, and set S... reg,ir S after registration with the predicted global affine transformation matrix reg,vi The second stage of the two-stage cascaded registration network architecture is input, namely, the RAFT-based elastic registration network. In each iteration of the elastic registration network, the updated coordinate grid is extracted and the initial absolute coordinate grid is subtracted to obtain the dense displacement field of the current iteration step. The dense displacement fields of all iteration steps are combined to form a multi-step residual displacement field sequence. Combining the registration inverse deformation field corresponding to the artificial deformation and the global affine transformation matrix predicted in the first stage, the corresponding residual displacement field between the two is calculated through spatial grid composite calculation as the true dense displacement field label to be fitted by the elastic registration network in this stage. The L1 distance between the two is calculated under the spatial weighting of the true registration effective mask. The overlapping foreground of the image is given a very high weight while the black border area of ​​the background is ignored. The L1 distance of each iteration step is weighted and summed as the multi-step prediction loss. At the same time, the second gradient of the final displacement field is calculated as the smoothness loss. The multi-step prediction loss and the smoothness loss are used to perform gradient backpropagation and end-to-end parameter fine-tuning on the elastic registration network to complete the training of the elastic registration network. The predicted joint dense spatial displacement field and the predicted registration effective mask are output. 52) Training of hierarchical fusion networks based on teacher-student knowledge distillation specifically includes the following steps: 521) Independent pre-training of the teacher hierarchical fusion network: The teacher hierarchical fusion network is set to be trained using the components after decoupling the original registered infrared and visible light images. The original registered infrared and visible light image pairs are decoupled to extract structural, texture and illumination features using the constructed physical heuristic feature decoupling mechanism. The features of each component are input into the initialized teacher hierarchical fusion network. Gradient backpropagation and end-to-end parameter fine-tuning of the teacher hierarchical fusion network are performed on the entire image using a hybrid fusion loss that includes pixel intensity, structural similarity and spatial gradient, thus completing the independent pre-training of the teacher hierarchical fusion network. 522) Construction of input features for student hierarchical fusion network: The cross-modal image training pair containing artificial deformation is decoupled using the constructed physical heuristic feature decoupling mechanism. The enhanced structural features of infrared and visible light are input into the pre-trained iterative global affine regression network and elastic registration network to obtain the predicted joint dense spatial displacement field and registration effective mask. The features of each component of visible light are resampled and registered using the predicted joint dense spatial displacement field. The registered features of each component of visible light and each component of infrared light are then jointly input into the student hierarchical fusion network to be optimized to output the fused image. 523) Generation of high-quality soft labels: In each training iteration, the original registered infrared and visible light image pairs are decoupled from the structure, texture and illumination features using the constructed physical heuristic feature decoupling mechanism. Each feature is then input into the frozen teacher layered fusion network. The high-quality fusion image output by the teacher layered fusion network, which is not affected by registration distortion and boundary artifacts, is used as a soft label. 524) Distillation Loss Calculation and Parameter Fine-tuning Based on Mask Intersection: Obtain the fused image output by the student hierarchical fusion network, and perform a logical AND operation between the real registration effective mask and the predicted registration effective mask to generate a mask intersection; within the area marked by the mask intersection, calculate the L1 distillation loss between the fused image output by the student hierarchical fusion network and the soft label output by the teacher hierarchical fusion network; combine the distillation loss with the hybrid fusion loss within the mask intersection area, and perform gradient backpropagation and end-to-end parameter fine-tuning on the student hierarchical fusion network. The trained student hierarchical fusion network is the hierarchical fusion network based on drawing logic.

6. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, enables the infrared and visible light image registration and fusion method based on physical heuristic feature decoupling as described in any one of claims 1-5.

7. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can implement the infrared and visible light image registration and fusion method based on physical heuristic feature decoupling as described in any one of claims 1-5.

Images