Method and model for determining registration homography of multi-modal images

Abstract

This disclosure relates to a method and model for determining the homography of multimodal images, belonging to the field of image registration technology. The method involves: embedding an initial noisy image into a diffusion denoising process to generate an initial intermediate image; using a cross-modal reference image as guidance, progressively denoising the initial intermediate image based on the diffusion denoising process to generate intermediate images at multiple intermediate steps; and performing complete denoising on the intermediate image at the current time step to generate a noise-free approximate image at the current time step; predicting the increment matrix at the current time step based on the cross-modal reference image and the noise-free approximate image at the current time step; selecting the increment matrix with the smallest registration error from the increment matrices at multiple time steps as the registration increment matrix, and determining the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix. The above method for determining the registration homography of multimodal images is used to determine the registration homography matrix.

Description

Technical Field

[0001] This disclosure relates to the field of image registration technology, and in particular to a method and model for determining the homography of multimodal images. Background Technology

[0002] Multimodal images refer to images captured by different sensors in the same scene, such as visible light (RGB) and depth image pairs, visible light and thermal infrared (TIR) ​​image pairs, and visible light and near-infrared (NIR) image pairs. Multimodal images can provide complementary information about targets and scenes, playing a crucial role in fields such as autonomous driving, medical diagnosis, and environmental monitoring. Due to the differences in imaging mechanisms among different sensors, multimodal images often cannot be directly aligned. Furthermore, some modal images are often captured under extreme physical conditions, inevitably introducing noise degradation. Image mismatch and noise interference pose significant obstacles to downstream tasks such as image fusion, object detection, and semantic segmentation. Therefore, designing an efficient multimodal registration method robust to noise to achieve accurate alignment of multimodal images has significant scientific and engineering value.

[0003] Image registration aims to align two images by fitting geometric differences such as rotation, translation, scaling, and perspective. Traditional image registration methods typically rely on hand-designed feature descriptors, such as SIFT, ORB, and KAZE, for feature detection, matching, and homography matrix computation. However, these methods often lead to inaccurate feature extraction in complex scenes, limiting their alignment performance. In recent years, deep learning-based methods have been proposed to address this issue, extracting image features and predicting geometric deformation matrix biases through autoencoders. Despite significant progress in single-modal image registration methods, challenges remain when handling cross-modal images. To address this, many researchers have developed multimodal image registration techniques, primarily including generative methods that convert one modality to another, or methods that map two modalities to a shared latent domain.

[0004] However, most existing methods for determining the homography of multimodal images are designed for sharp images and ignore the impact of noise degradation, which greatly limits their applicability in complex real-world environments. More importantly, simple joint denoising cannot directly guarantee improved image registration accuracy. Therefore, how to improve the registration accuracy of multimodal images is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned technical issues, this disclosure provides a method and model for determining the homography of multimodal images, as well as a method and model for multimodal image registration.

[0006] On one hand, this disclosure provides a method for determining the registration homography of a multimodal image. This method includes: embedding an initial noisy image into a diffusion denoising process to generate an initial intermediate image; progressively denoising the initial intermediate image based on the diffusion denoising process, guided by a cross-modal reference image, to generate intermediate images at multiple intermediate steps; and performing complete denoising on the intermediate image at the current time step to generate a noise-free approximate image at the current time step. In multiple time steps, based on the cross-modal reference image and the noise-free approximate image at the current time step, predicting the increment matrix for the current time step; selecting the increment matrix with the smallest registration error among the increment matrices of the multiple time steps as the registration increment matrix, and determining the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

[0007] On the other hand, this disclosure provides a registration method for multimodal images, which includes the registration homography determination method for multimodal images provided in some of the above embodiments, and registration of an initial noisy image and a cross-modal reference image based on the registration homography matrix.

[0008] In another aspect, this disclosure provides a registration homography determination network for multimodal images, which includes: an embedding subnetwork, a multimodal image denoising subnetwork, a multimodal homography estimation subnetwork, and an adaptive homography selection subnetwork. The embedding subnetwork is used to: embed the initial noisy image into the diffusion denoising process to generate an initial intermediate image; the multimodal image denoising subnetwork is used to: use the cross-modal reference image as a guide to perform stepwise denoising on the initial intermediate image based on the diffusion denoising process to generate intermediate images of multiple intermediate steps, and to perform complete denoising on the intermediate image of the current time step to generate a noise-free approximate image of the current time step; the multimodal homography estimation subnetwork is used to: predict the increment matrix and homography matrix of the current time step based on the cross-modal reference image and the noise-free approximate image of the current time step in multiple time steps; the adaptive homography selection subnetwork is used to: select the increment matrix with the smallest registration error among the increment matrices of multiple time steps as the registration increment matrix, and determine the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

[0009] Furthermore, this disclosure provides a multimodal image registration model, which includes a multimodal image registration homography determination network and a registration execution subnet provided in some of the above embodiments. The registration execution subnet is used to: register an initial noisy image and a cross-modal reference image according to the registration homography matrix.

[0010] The technical solution provided in this disclosure has the following advantages compared with the prior art: Using a cross-modal reference image as prior guidance, the denoising subnetwork learns the spatial correspondence between the intermediate image and the cross-modal reference image during the denoising process. This results in noise-free approximate images that are not only visually clean, but more importantly, their spatial structure is highly consistent with that of the cross-modal reference image. This reduces the negative impact of noise interference on homography estimation from the source, establishing the registration task on a high-quality, feature-aligned noise-free approximate image, significantly improving registration accuracy and stability.

[0011] Furthermore, selecting the incremental matrix with the smallest registration error among multiple time steps as the registration incremental matrix can ensure the optimal registration accuracy of the registration incremental matrix and avoid unsatisfactory registration results due to errors in a single time step or local optima.

[0012] Furthermore, the homography matrix corresponding to the time step of the registration increment matrix is ​​determined as the registration homography matrix, so that the final homography matrix used for image alignment has the globally optimal geometric transformation accuracy, improving the registration effect between noise-free approximate images and cross-modal reference images, thereby enhancing the robustness of the overall registration process and effectively suppressing error accumulation, disturbances and local extrema problems that occur during the iteration process. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0014] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a structural block diagram of the multimodal image registration homography determination network provided in the embodiments of this disclosure; Figure 2 Another structural block diagram of the multimodal image registration homography determination network provided in the embodiments of this disclosure; Figure 3 This diagram illustrates the steps of the registration homography determination method for multimodal images provided in this embodiment of the disclosure. Figure 4 for Figure 2 The structural block diagram of the multimodal denoising subnet in the image; Figure 5This is a structural block diagram of the multimodal homography estimation subnet provided in the embodiments of this disclosure; Figure 6 This is a structural block diagram of the adaptive homography selection subnet provided in the embodiments of this disclosure; Figure 7 The training method for the multimodal image registration homography determination network provided in this embodiment of the disclosure includes a graph showing the correspondence between the degree of distortion and the correlation graph. Figure 8 This is a flowchart illustrating the training method for a multimodal image registration homography determination network provided in this embodiment of the disclosure. Detailed Implementation

[0016] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0017] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0018] Multimodal image registration refers to aligning two images from different modalities to eliminate spatial geometric differences. However, due to factors such as shooting equipment, exposure time, and imaging environment, multimodal images not only differ in semantic information but are also often contaminated by noise, which hinders the application of multimodal image registration algorithms in real-world scenarios.

[0019] This disclosure proposes a multimodal image registration method and model based on a diffusion model, namely the HomoDiff method and model, to solve the problem of multimodal image alignment under noise interference. In the HomoDiff method and model, this disclosure designs a joint task network for denoising and registration, namely, including a multimodal image denoising subnetwork. and multimodal homography estimation subnet Noise is progressively eliminated through a diffusion inference process, and the homography increment matrix is ​​predicted. This homography matrix is ​​then used to describe the geometric deformation between image pairs. (Multimodal image denoising subnetwork) Noise removal is performed under the guidance of a cross-modal reference image, while a multimodal homography estimation subnetwork... The corresponding increment matrix is ​​predicted under the current denoising intensity. To select the optimal alignment scheme from the series of results generated during the inference process, this disclosure further proposes an adaptive selection subnet. This is used to evaluate the alignment error corresponding to the increment matrix at different time steps. Among them, the HomoDiff method and model can achieve higher alignment accuracy and stronger noise robustness than existing methods in the registration of multiple multimodal images.

[0020] The HomoDiff method provided in this disclosure includes an incremental matrix estimation stage and a multimodal image denoising stage.

[0021] The following describes the registration homography determination method and network for multimodal images provided in this disclosure.

[0022] Figure 1 This is a block diagram of the multimodal image registration homography determination network provided in the embodiments of this disclosure. Figure 2 Another structural block diagram of the multimodal image registration homography determination network provided in the embodiments of this disclosure; Figure 3 This diagram illustrates the steps of a method for determining the registration homography of multimodal images provided in an embodiment of this disclosure.

[0023] Please see Figures 1 to 3 This disclosure provides a method for determining the registration homography of multimodal images. The method includes steps S1 to S4. The method can be implemented based on a multimodal image registration homography determination network, which may include an embedding subnetwork, a multimodal image denoising (MID) subnetwork, a multimodal homography (MHE) subnetwork, an adaptive homography selection (AHS) subnetwork, and a registration execution subnetwork. In this application, the multimodal image registration homography determination model can also be referred to as a Diffusion Model.

[0024] The embedded subnet is used to: perform step S1. S1, embed the initial noisy image into the diffusion denoising process to generate an initial intermediate image.

[0025] Due to the influence of factors such as illumination, temperature, current, and quantization, visible light mode images inevitably contain noise. (Initial noisy image) It could be a distorted, noisy visible light image, where the initial noisy image... It can be understood as a natural noise image.

[0026] Cross-modal reference image It can be a fixed image of another modality with low noise pollution.

[0027] In related technologies, when ordinary diffusion models are used for image restoration, they usually start denoising inference from images with pure random noise, which introduces high uncertainty into image denoising and leads to instability in homography estimation (registration).

[0028] The registration homography determination method for multimodal images provided in this disclosure starts from a natural noise image (i.e., the initial noise image). (Start denoising inference, rather than using a purely random noise image.)

[0029] The diffusion denoising process can be implemented based on DDPM (denoising diffusion probabilistic models). This process has two core stages: a forward process and a reverse process. The forward process involves gradually adding noise to a real image until it becomes a purely noisy image. The reverse process, which can be understood as the denoising process, starts from the purely noisy image and gradually removes the noise to restore the real image.

[0030] Among them, the initial noise image In the process of embedding diffusion denoising, it can be understood as transforming the initial noisy image... As the starting point for the reverse process, it enters the computational flow of progressive denoising.

[0031] In step S1, intermediate images at multiple time steps are obtained based on the diffusion denoising process.

[0032] Wherein, the larger the time step t, the lower the denoising intensity corresponding to that time step, and the intermediate state image at that time step t. The higher the noise level, the better the intermediate state image at time step t. The closer the noise is to the initial noise image The initial noise level.

[0033] The smaller the time step t, the better the intermediate state image at that time step t. The less noise there is, the higher the noise reduction strength.

[0034] In step S1, an initial intermediate image can be generated based on the initial noisy image using the diffusion denoising process. .

[0035] In some embodiments, step S1 above includes steps S11 and S12 below. The embedded subnet includes a third computing unit and an embedding unit.

[0036] The third calculation unit is used to execute step S11. S11: Analyze the initial noise level of the initial noise image and determine the time step with the smallest absolute difference from the initial noise level during the diffusion denoising process as the preset time step.

[0037] Due to the initial noisy image This is a naturally noisy image, so we need to first extract this real, initial noisy image. Instead of starting from zero, it is incorporated into the preset time step T of the diffusion denoising process.

[0038] In some embodiments, in step S11, the preset time satisfies the following formula (1).

[0039] Formula (1): .

[0040] Where T is the preset time step. The initial noise level, The cumulative noise dispatch coefficient is denoted as ; where satisfy: ,in, This is the noise schedule for the current time step. These are a series of hyperparameters predefined before training.

[0041] This equals all steps from step 1 to step T. The product of.

[0042] And representing the multimodal image denoising subnetwork At time step The noise reduction capability. The preset time step T is determined to be... and The time step with the smallest absolute difference.

[0043] For example, in step S11, iterate through all time steps t (e.g., t=1,2,...,1000) and calculate the theoretical noise level corresponding to each time step t. Then, the theoretical noise level and the initial noise image are calculated for each time step. The absolute difference of the initial noise level σ is calculated; then, the time step with the smallest difference is selected as the final preset time step T, where the preset time step T is the starting time step that best matches the initial noise level during the diffusion denoising process. With initial noise level The increase in noise level indicates that for initial noisy images with high noise levels, this increases. A denoising method with deeper denoising inference steps, i.e., undergoing more denoising time steps, should be adopted.

[0044] In related technologies, ordinary diffusion models are fixed to start denoising from the maximum time step, if the input initial noisy image... If the noise level is low, starting denoising from the maximum time step will lead to over-denoising and loss of image details.

[0045] In this disclosure, in step S11, based on the initial noise image Obtain the initial noise level Match the embedding time step with the smallest absolute difference (i.e., the preset time step), and make the starting time step of the diffusion denoising process (i.e., the preset time step T) match the initial noise image. This allows for adaptation to the actual noise level, which can then be used to generate intermediate images at multiple time steps. This lays a stable foundation. In addition, in practical applications, the noise levels of multimodal images (such as infrared / visible light) vary greatly; for example, lighting, equipment, and environment can all affect the initial noisy image. Obtain the initial noise level The step S11 provided in this application enables the registration homography determination method for multimodal images to adapt to initial noise images with different noise levels. Instead of only adapting to images with a fixed noise level, the registration homography determination method for multimodal images provided in this disclosure can be adapted to different noise scenarios.

[0046] The embedding unit is used to perform step S12. S12: embed the initial noise image into a preset time step of the diffusion denoising process, and generate an intermediate image of the preset time step based on the initial noise image.

[0047] This disclosure will use the initial noise image Embedded into the diffusion denoising process to generate an initial intermediate state image. That is, the intermediate state image at a preset time step. .

[0048] Among them, the intermediate state image at the preset time step and initial noisy image The relationship between them can be described by the following formula (2).

[0049] Formula (2): .

[0050] Among them, the initial noise image Multiply The distribution is transformed into the distribution required for the preset time step T of the diffusion denoising process.

[0051] Multimodal image denoising subnet Used to perform step S2. S2: Guided by the cross-modal reference image, the initial intermediate image is progressively denoised based on the diffusion denoising process to generate intermediate images of multiple intermediate steps, and the intermediate image of the current time step is completely denoised to generate a noise-free approximate image of the current time step.

[0052] Among them, the intermediate state image at the current time step Complete denoising refers to starting from time step T and gradually progressing towards t=0 to denoise the image, ultimately generating a noisy approximation. .

[0053] Among them, cross-modal reference image As a priori guide, the multimodal image denoising subnet Learning intermediate images during denoising With cross-modal reference image The spatial correspondence between them. This makes the generated noise-free approximate image... Not only is it visually clean, but more importantly, it's a noise-free approximation image. Spatial structure and cross-modal reference image The spatial structure consistency is high. This reduces the negative impact of noise interference on homography estimation from the source, allowing the registration task to be built on high-quality, feature-aligned, noise-free approximate images. Based on this, the registration accuracy and stability have been significantly improved.

[0054] Step S2 may include the following steps S21 to S22, multimodal image denoising subnetwork. It includes a noise estimation unit, a complete denoising unit, and a first calculation unit.

[0055] The noise estimation unit is used to perform step S21. S21: In multiple time steps, guided by the cross-modal reference image and combined with the noise level of the current time step, the noise value of the intermediate image of the current time step is estimated.

[0056] Among them, the multimodal image denoising subnetwork At each time step Predicted noise The above step S21 can be expressed by the following formula (3).

[0057] Formula (3): .

[0058] Among them, the intermediate state image estimated at each time step The noise values ​​are different.

[0059] The time step t directly encodes the current noise level; that is, the larger the time step t, the higher the intermediate image estimated by the noise estimation unit. The louder the noise.

[0060] In step S21, using a cross-modal reference image The provided prior structure effectively constrains the noise estimation process, enabling the model to accurately preserve intermediate images while identifying noise. It extracts key structural and detailed information, effectively suppresses artifacts, oversmoothing, or structural distortion in high-noise areas, and improves the visual quality and structural integrity of the denoised image.

[0061] By estimating intermediate state images at each time step In the process of calculating the noise value, the noise level at the current time step is fused with the cross-modal reference image. The guidance ensures that the noise value estimation results show a stable and orderly convergence trend as the time step decreases, avoiding the amplification of the estimation error of a single time step in multiple iterations, and significantly enhancing the robustness and predictability of the entire diffusion denoising process.

[0062] The complete denoising unit is used to execute step S22. S22: Based on the noise value of the intermediate state image at the current time step, the intermediate state image at the current time step is completely denoised to obtain a noise-free approximate image at the current time step.

[0063] In step S22, based on the intermediate image obtained in step S21 noise value Effectively removes intermediate state images To obtain a high-quality, low-distortion noise-free approximate image by eliminating noise components in the image. This provides a reliable data foundation for estimating homography matrices and geometric transformation parameters in subsequent registration subnets, significantly improving the accuracy and stability of registration results, and thus enhancing the final performance of overall image registration, reconstruction, or fusion tasks.

[0064] In step S22, at each time step, a noise-free approximate image is generated. and intermediate state images noise value It satisfies the following formula (4).

[0065] Formula (4): .

[0066] in, This represents the noise value at the current time step.

[0067] Figure 4 for Figure 2 The structural block diagram of the multimodal denoising subnet in the diagram.

[0068] Please see Figure 4In some embodiments, step S21 includes the following steps S211 and S212. The noise estimation unit includes a cross-modal guidance module and a denoising U-Net module. Multimodal image denoising subnet It consists of two parts: a cross-modal guidance module and a denoising U-Net module. The cross-modal guidance module is used to generate features for multimodal guidance, and the denoising U-Net module is used to predict the noise at the current time step. Predicted noise It was then used to calculate a clean image approximation. and pre-intermediate image .

[0069] The cross-modal guidance module is used to execute step S211, which involves extracting feature information from the cross-modal reference image and generating a cross-modal guidance term based on the noise level of the current time step.

[0070] In the cross-module guidance module, there are three guidance layers 1, each serving as the coarsest-grained guidance. Each guidance layer 1 includes two residual blocks and one downsampled convolution. Guidance layers 1 are used to extract multi-scale coarse-grained cross-modal guidance features from the reference image y. The three guidance layers 1 output G11, G12, and G13 sequentially. G11 and G12, after being adapted to zero convolution (ZC) dimension, are concatenated with the corresponding layer features of the denoising U-Net module, providing coarse-grained global semantic guidance for the denoising process, constraining the overall scene layout and main structure of the noisy image, and avoiding large-scale structural distortions after denoising. G13 is a feature that has undergone secondary downsampling, serving as the input to guidance layer 2, providing the downsampled semantic basis for subsequent fine feature extraction.

[0071] There are two guidance layers 2, which consist of residual blocks, attention blocks, and downsampling convolutions. Guidance layer 2 takes the downsampling feature G13 output from guidance layer 1 as input and is composed of residual blocks, attention blocks, and downsampling convolutions. The two guidance layer 2 units output G21 and G22 sequentially. G21 is concatenated with intermediate layer features of the denoising U-Net after zero convolution, strengthening the association between key regions of the reference image and the noisy image through a cross-modal attention mechanism, providing medium-granular local detail guidance for the denoising process, focusing on repairing region edges and texture contours. G22 is a feature that has been downsampled again, serving as input to guidance layer 3, providing the downsampling basis for the highest precision detail extraction.

[0072] There are two guidance layers 3, consisting of residual blocks and attention blocks. The two guidance layers 3 output G31 and G32 sequentially. G31 is concatenated with the shallowest features of the denoising U-Net after zero convolution, providing fine-grained and high-precision detail guidance for the denoising process, focusing on subtle textures and high-frequency information, and accurately repairing the blurring and loss of details caused by noise; G32 is the final output of the module, representing the top-level semantic features extracted by the cross-modal guidance module.

[0073] In the guidance layers 1, 2, and 3, the convolutions are 3×3 convolutions, and the downsampling convolutions are 2× downsampling convolutions. The residual blocks are residual linking blocks, the attention blocks are cross-modal attention blocks, ZC is a zero convolution block, and C represents concatenation.

[0074] The cross-modal guidance module will use cross-modal reference images and time step As input, the time step can indicate the noise level, thus assisting the cross-modal guidance module in working with the cross-modal reference image. Feature information is extracted from the reference image. The cross-modal guidance module consists of a series of residual blocks and attention blocks, followed by several zero-convolutional layers. The residual blocks and attention blocks extract features from the reference image and integrate the features through skip connections. The data is passed to the zero convolutional layer, which is... Convolution is used to project the guiding features onto the denoising U-Net module. All outputs from the zero-convolutional layer are concatenated to form the cross-modal guiding term. .

[0075] The denoising U-Net module is used to perform step S212, which extracts hierarchical features from the intermediate image of the current time step and predicts the prediction noise of the current time step under the guidance of the cross-modal guidance term.

[0076] The denoising U-Net module receives cross-modal guidance terms. Time step and intermediate state images As input, it is used to predict the intermediate state image at the current time step t. noise value The denoising U-Net module consists of an encoder and a decoder, which are constructed from a series of residual blocks and attention blocks, following a widely adopted diffusion network structure. The encoder obtains data from intermediate images. Hierarchical features are extracted from the decoder, while the decoder extracts hierarchical features from the decoder. Guided by these features, noise values ​​are reconstructed. The encoding and decoding process in the denoising U-Net module can be formulated as shown in the following formula (5).

[0077] Formula (5): .

[0078] in, This represents the concatenation operator.

[0079] Multimodal homography estimation subnet Used for: performing step S3. S3, predicting the increment matrix of the current time step based on the cross-modal reference image and the noise-free approximation image of the current time step across multiple time steps.

[0080] Among them, the incremental matrix A noise-free approximation of the current time step With cross-modal reference image The slight geometric deviations that are not yet aligned are stepwise correction terms to the overall homography matrix.

[0081] This disclosure integrates the denoising process and homography estimation into a joint task network. Multimodal denoising subnetwork At each time step Calculate noise-free approximate image Multimodal homography estimation subnet Based on the noise-free approximate image and cross-modal guidance images Predicted Increment Matrix Multimodal homography estimation subnet Used to calculate the increment matrix Among them, the multimodal homography estimation subnet The core mathematical expression formula is shown in the following formula (6).

[0082] Formula (6): .

[0083] Figure 5 The multimodal homography estimation subnet provided in the embodiments of this disclosure The structural block diagram.

[0084] Please see Figure 5 In some embodiments, step S3 includes: steps S31 and S32 described below. Multimodal homography estimation subnet It includes a multi-scale feature extraction unit and a cross-modal homography estimation unit.

[0085] The multi-scale feature extraction unit is used to: execute step S31. S31: Extract multiple first-scale feature maps from the noise-free approximate image at the current time step to form a first multi-scale feature set, and extract multiple second-scale feature maps from the cross-modal reference image to form a second multi-scale feature set.

[0086] Step S31 can be the first stage of step S3, namely the multi-scale feature extraction stage.

[0087] For example, in the first stage, two convolutional branches with shared parameters are used to extract noise-free approximation images. Extract three primary feature maps from coarse to fine. Where 1 ≤ n ≤ 3. The three first feature maps constitute the first multi-scale feature set. .in, noise-free approximate image coarse-grained map, , noise-free approximation image Fine-grained feature map.

[0088] From cross-modal reference image Extract three secondary feature maps from coarse to fine. Three second feature maps Constructing the second multi-scale feature set .in, For cross-modal reference images coarse-grained map, , For cross-modal reference images Fine-grained feature map.

[0089] First Multi-Scale Feature Set Second multi-scale feature set Used to support the second-stage incremental matrix estimation process.

[0090] The cross-modal homography estimation unit is used to: execute step S32. S32: Based on the first multi-scale feature set and the second multi-scale feature set, determine the offset between the noise-free approximate image and the cross-modal reference image at the current time step, and determine the increment matrix at the current time step based on the offset.

[0091] In step S32, based on the noise-free approximate image First multi-scale feature set and cross-modal reference image The three second multi-scale feature sets Calculate the offset of the current time step.

[0092] The offset is used to characterize the noise-free approximate image. and cross-modal reference image The coordinate difference between them can directly reflect the noise-free approximate image. and cross-modal reference image The actual positional differences between them.

[0093] In this disclosure, the first multi-scale feature set can be fully utilized. Second multi-scale feature set The system incorporates global structural information and local detail information, ensuring matching stability and avoiding mismatches under the constraints of coarse-grained features. Simultaneously, it improves the accuracy of offset calculation with the assistance of fine-grained features, enabling the determined offsets to more accurately and robustly reflect the noise-free approximate image. and cross-modal reference image The actual geometric positional differences between them provide reliable geometric constraints for solving the subsequent incremental matrix, thereby improving the accuracy and stability of image registration.

[0094] In some embodiments, step S32 includes the following steps S321 to S324. The cross-modal homography estimation unit includes: a correlation calculation module, an upsampling module, a decoding module, and an incremental matrix estimation module.

[0095] The correlation calculation module is used to perform step S321. S321: Determine the cosine similarity at multiple scales between the first position in the noise-free approximate image and the second position in the cross-modal reference image, based on the first multi-scale feature set and the second multi-scale feature set.

[0096] In step S321, at each corresponding scale n, a noise-free approximate image is calculated. The cosine similarity between the first position p in the first feature map and the second position u in the feature map of the cross-modal reference image.

[0097] Finally, the cosine similarity at multiple scales is obtained, which is used to describe the degree of feature matching between the first position p and the second position u at different granularities.

[0098] For example, if the first multi-scale feature set includes three first feature maps and the second multi-scale feature set includes three second feature maps, then in step S321, three sets of cosine similarities can be obtained, corresponding to matching information at three granularities: fine, medium, and coarse. For example, the three sets of cosine similarities are respectively... , and .

[0099] In step S321, the cosine similarity satisfies the following formula (7).

[0100] Formula (7): .

[0101] in, This represents the noise-free approximate image at the nth scale. The cosine similarity between the first position p in the image and the second position u in the cross-modal reference image. It can be abbreviated as . Represents a noise-free approximate image The feature vector at the first position p on the nth scale feature map. This represents the feature vector at the second position u on the nth scale feature map of the cross-modal reference image. This is the L2 norm operator, which calculates the Euclidean length (modulus) of the eigenvector.

[0102] In some embodiments, the second position U is located at the first position. Within the neighborhood of a square window with center r and radius r as the first preset radius.

[0103] in, ,in, This indicates that all elements within the tensor are real numbers. Let p represent the height and width of the nth scale feature map, respectively, corresponding to all positions p on the noise-free approximation image feature map (i.e., the first feature map). The nth scale feature map includes the noise-free approximation image. The feature map at the nth scale and the feature map of the cross-modal reference image at the nth scale. This represents the total number of candidate second positions u within a square neighborhood of radius r around each first position p, where the window side length is... There are a total of 2 candidate positions u. One position.

[0104] In this method, by limiting the search space to a square neighborhood with a radius of a first preset radius r, the search space is compressed from global to local. This significantly reduces the number of feature matching calculations, effectively lowering the model's inference time and meeting the needs of real-time or lightweight applications. Simultaneously, it avoids global mismatches to ensure registration accuracy and enhances the algorithm's robustness and practicality.

[0105] The upsampling module is used to execute step S322. S322: Upsample the cosine similarity at multiple scales to obtain a spatial mismatch correlation map between the first position and the second position at multiple scales.

[0106] In step S322, the spatial mismatch correlation diagram Used to characterize a noise-free approximate image in the original image coordinate system. The overall matching degree distribution between each position and the corresponding position in the cross-modal reference image provides a unified geometric basis for subsequent accurate positioning of the four-point offset.

[0107] For example, in step S322, the cosine similarity is calculated at three different scales. , and Upsampling yields spatial mismatch correlation maps at three different scales, which are as follows: , .

[0108] Cosine similarity Spatial mismatch correlation diagram after upsampling Size and noise-free approximate image Consistency, thereby enabling noise-free approximation of images Calculate noise-free approximate images at the pixel level. and cross-modal reference image The offset between them, and the resulting increment matrix can also be directly applied to the noise-free approximation image. Geometric transformations.

[0109] In step S322, the spatial mismatch correlation diagram at multiple scales is generated. It satisfies the following formula (8).

[0110] Formula (8): .

[0111] In formula (8), As an upsampling method, the upsampling factor is , This is a spatial mismatch correlation map used to represent a noise-free approximate image. The nth scale feature and cross-modal reference image The nth scale feature Spatial mismatch between them, spatial mismatch correlation diagram It can also be called a correlation map.

[0112] The decoding module is used to execute step S323. S323: Decode the spatial mismatch correlation map at multiple scales into the offset of the current time step.

[0113] In step S323, this disclosure may employ an offset decoder (OD) to analyze the spatial mismatch correlation graph. Decoded as offsets. For example, at different time steps t, this disclosure uses a coarse-to-fine approach from the set... Select spatial mismatch correlation diagram Perform offset prediction and The data is fed into OD (offset decoder) to decode the offset.

[0114] For example, in step S323, the acquired offsets include the coordinate offsets of at least four sets of non-collinear matching point pairs. In one matching point pair, one point comes from a noise-free approximation image. Another point comes from the cross-modal guidance image. .

[0115] The formula for calculating the offset in step S323 is as follows (9).

[0116] Formula (9): .

[0117] in, Indicates the current time step The offset below.

[0118] In step S323, the spatial mismatch correlation diagram is processed. , and Perform offset prediction to obtain the four-point offset.

[0119] The incremental matrix estimation module is used to execute step S324. S324: Based on the direct linear transformation algorithm, calculate the incremental matrix of the current time step according to the offset of the current time step.

[0120] The calculation formula in step S324 is as follows: formula (10).

[0121] Formula (10): .

[0122] In steps S321 to S324 provided in this disclosure, the cosine similarity between the first position P and the second position U is calculated through multi-scale features. This improves the robustness of cross-modal feature matching and reduces complexity through local neighborhood calculation, providing a reliable mismatch basis for subsequent offset decoding. Furthermore, the multi-scale cosine similarity is upsampled and uniformly mapped to the original image coordinate system to eliminate scale differences and generate a multi-scale spatial mismatch correlation map. This supports registration optimization from coarse to fine. It integrates multi-scale spatial mismatch correlation diagrams. End-to-end decoding yields the current time step offset, and the output contains at least four sets of non-collinear matching point pairs to ensure global offset consistency and provide sufficient geometric constraints for the DLT (Direct Linear Transformation) algorithm.

[0123] Through the above steps S1 to S3, this disclosure obtains a set of candidate incremental matrices. Candidate Increment Matrix Incremental matrix including multiple time steps .

[0124] To select the optimal matrix, i.e., the registration increment matrix, from these candidates, this disclosure further proposes an Adaptive Homography Selection (AHS) subnet. This AHS subnet is used to estimate the registration error of each candidate matrix and select the increment matrix that produces the minimum registration error. As the registration increment matrix .

[0125] In some examples, both the first and second feature extraction units can include CNR (Convolution-Normalization-ReLU), two cascaded base blocks, and a basic block. The CNR consists of a cascade of convolution, regularization, and ReLU (Rectified Linear Unit). The convolution is a 3×3 convolution. The basic block consists of two residual connection structures formed by the CNR module and the convolution, with a convolution at the end. The convolution is a 1×1 convolution.

[0126] The cross-modal homography estimation unit includes a correlation calculation module, an upsampling module, and an offset decoder (OD).

[0127] Next, this disclosure will describe how the Adaptive Homography Selection (AHS) subnet predicts the incremental matrix. Registration error .

[0128] Figure 6 This is a structural block diagram of the adaptive homography selection subnet provided in the embodiments of this disclosure.

[0129] Please see Figure 6 Adaptive homography selection subnet Used for: Executing step S4. S4: Among the increment matrices of multiple time steps, select the increment matrix with the smallest registration error as the registration increment matrix, and determine the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

[0130] Adaptive homography selection subnet The increment matrix used to predict each time step Registration error Registration error The smaller, the more it means The higher the registration accuracy, the better.

[0131] Among them, the registration error is selected in multiple time steps. Minimum increment matrix As the registration increment matrix This ensures the registration increment matrix. This ensures optimal registration accuracy and avoids suboptimal registration results caused by errors in a single time step or local optima.

[0132] And the registration increment matrix The homography matrix corresponding to the time step is determined as the registration homography matrix. This makes the final registration homography matrix used for image alignment... It achieves globally optimal geometric transformation accuracy, improving noise-free approximate images. The registration effect with the cross-modal reference image y is improved, thereby enhancing the robustness of the overall registration process and effectively suppressing error accumulation, disturbances and local extrema problems that occur during the iteration process.

[0133] Step S4 may include the following steps S4.1 and S4.2.

[0134] S4.1 Among the increment matrices of multiple time steps, select the increment matrix with the smallest registration error as the registration increment matrix.

[0135] In step S4.1, the index of the minimum error can be determined using the following formula (23). .in, The time step that minimizes the registration error.

[0136] Formula (23): .

[0137] Among them, the adaptive homography selection subnetwork This invention searches for the registration error at each time step. Select the time step with the smallest error from all results. Correspondingly This is the final increment matrix used for registration.

[0138] S4.2. Determine the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

[0139] Step S4.1 may include steps S41 to S47 below, and step S4.2 may include step S47 below. Adaptive homography selection subnet It may include a first feature extraction unit, a second feature extraction unit, a coordinate flow generation unit, an alignment unit, an error estimation unit, a registration increment matrix determination unit, and a homography matrix determination unit.

[0140] The first feature extraction unit is used to perform step S41, which is to obtain the first depth feature map of the current time step from the noise-free approximate image of the current time step.

[0141] The second feature extraction unit is used to perform step S42, which is to obtain a second depth feature map from the cross-modal reference image.

[0142] Considering that the increment matrix is ​​decoded from features, this disclosure evaluates its alignment quality within the feature domain. To this end, this disclosure employs a symmetrical dual-branch architecture to extract the alignment quality from the noise-free approximation image. and cross-modal reference image Extract features from them.

[0143] The symmetrical dual-branch architecture can include a first feature extraction unit and a second feature extraction unit.

[0144] Among them, the noise-free approximation image The obtained first depth feature map is represented as .

[0145] And from the cross-modal reference image The second depth feature map is represented as .

[0146] The coordinate stream generation unit is used to execute step S43. S43: Generate the coordinate stream for the current time step based on the increment matrix of the current time step using coordinate stream encoding.

[0147] In step S43, this disclosure utilizes the widely used coordinate flow coding (CFC) from the incremental matrix Generate a coordinate stream.

[0148] The alignment unit is used to perform step S44. S44: Using the coordinate flow of the current time step, the first depth feature map of the current time step is warped to obtain the first aligned version map of the current time step.

[0149] In step S44, the twisting operation is formulated as shown in the following formula (11).

[0150] Formula (11): .

[0151] in, This is the first aligned version of the image. Represents the distortion operator. Downsampling operator. Used to adjust the resolution of the coordinate flow to match the first depth feature map Matching the resolution, This indicates the use of CFC to adjust the increment matrix. The coordinate stream generated by encoding.

[0152] The error estimation unit is used to perform step S45. S45: Estimate the registration error between the first aligned version map and the second depth feature map of the current time step, as the registration error of the incremental matrix of the current time step.

[0153] In step S45, this disclosure uses the first aligned version diagram after alignment. Second depth feature map To estimate the registration error To evaluate alignment accuracy, this disclosure first calculates a first aligned version of the graph. With the second depth feature map Spatial mismatch correlation graph based on cosine similarity The spatial mismatch correlation diagram Then with time steps The location codes are fed together into the regression module (Reg) of the error estimation unit to predict the registration error. Among them, registration error It satisfies the following formula (12).

[0154] Formula (12): .

[0155] in, This indicates the position code.

[0156] The regression module Reg is a CNN-based network consisting of several convolutional layers followed by a fully connected layer. During inference, this process is performed at each time step. Repeat the process and output a series of registration errors. .

[0157] In some examples, both the first and second feature extraction units may include a CNR and a base block, where the CNR consists of a 3×3 convolution, regularization, and ReLU. The base block consists of a convolution, regularization, and ReLU. The coordinate flow generation unit internally integrates offset encoding.

[0158] The error estimation unit may include a second correlation module, a second upsampling module, and a regression block, wherein the second correlation module is used to calculate the first aligned version graph. With the second depth feature map Cosine similarity between them.

[0159] The second upsampling module is used to process the first aligned version image. With the second depth feature map The cosine similarity between the two is upsampled to obtain the first aligned version image. With the second depth feature map Spatial mismatch correlation diagram .

[0160] The spatial mismatch correlation diagram Then with time steps The location codes are fed together into the regression module (Reg) of the error estimation unit to predict the registration error. .

[0161] The registration increment matrix determination unit is used to execute step S46. S46: Among the increment matrices of multiple time steps, the increment matrix with the smallest registration error is selected as the registration increment matrix.

[0162] In step S46, the incremental matrix is ​​registered. It satisfies the following formula (13).

[0163] Formula (13): .

[0164] In formula (13), And t∈[1,T], which means that in step S46, all T time steps are traversed. Let argmint∈[1,T]et be the index of the selected optimal time step. Let argmint∈[1,T]et represent the time step t that minimizes the registration error et among all time steps, denoted as .

[0165] The homography matrix determination unit is used to execute step S47. S47: Determine the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

[0166] The registration homography matrix can be obtained from the registration increment matrix. .

[0167] In steps S41 to S47, the noise-free approximate image is obtained. The first and second depth feature maps are obtained from the cross-modal reference image, respectively. A symmetrical dual-branch architecture is used to ensure feature space consistency. Alignment quality is evaluated within the feature domain to avoid interference from modal differences in the original image, providing a stable feature basis for subsequent registration error calculation. The original image for the first depth feature map refers to a noise-free approximate image. The original image of the second depth feature map refers to the cross-modal reference image.

[0168] By using coordinate flow to spatially warp the first depth feature map, a first aligned version image is obtained. This completes the geometric alignment of the approximate image and the reference image within the feature domain, intuitively reflecting the registration effect of the current incremental matrix and providing directly comparable alignment features for error assessment.

[0169] The registration error between the first aligned version image and the second deep feature map is estimated, and the matching difference in the feature domain is quantified as the registration error of the current incremental matrix. This enables an accurate and objective assessment of the alignment quality of the incremental matrix, avoiding bias caused by subjective judgment.

[0170] Among the increment matrices at multiple time steps, the one with the smallest registration error is selected as the registration increment matrix, and the homography matrix at the corresponding time step is determined as the registration homography matrix. By using a multi-time-step iterative optimization mechanism, the problem of getting stuck in local optima or error accumulation in a single iteration can be avoided, and the final registration matrix can be guaranteed to have globally optimal geometric alignment accuracy. In addition, it can enhance the robustness and stability of the cross-modal image registration process and improve the registration reliability in real-world scenarios.

[0171] In some embodiments, in step S47, the homography matrix of the current time step satisfies the following formula (14).

[0172] Formula (14): .

[0173] in, Let be the homography matrix at the current time step t; The homography matrix for the next time step t+1 .

[0174] Among them, the registration increment matrix The corresponding homography matrix is ​​the registration homography matrix. .

[0175] in, ,in, Let I be the homography matrix at time step T+1. Let I be the identity matrix.

[0176] In some embodiments, the registration homography determination method for multimodal images may further include step S5 below. The multimodal image registration homography determination network may further include an iterative computation subnet, which is used to perform step S5. S5, based on cross-modal reference images The noise level at the current time step determines the intermediate state image of the previous time step t-1. The probability distribution in DDPM, where the intermediate state image at the current time step. Given a cross-modal reference image y and time step t, ideally, the intermediate image of the previous time step... The distribution follows a pattern. For the mean, The variance follows a Gaussian distribution. Specifically, in step S5, the intermediate state image... The distribution of satisfies the following formula (15).

[0177] Formula (15): .

[0178] In equation (15), Represents the ideal probability distribution. This represents a Gaussian distribution (normal distribution). Indicates the noise mean. This represents the noise variance at the current time step. I represents the noise standard deviation at the current time step, and I is the identity matrix.

[0179] Among them, the intermediate state image estimated at the current time step noise value It can be used as the formula (15) .

[0180] Step S5 may include steps S51 and S52 as described below. The iterative computation subnet may include a first computation unit and a second computation unit.

[0181] The first computing unit is used to execute step S51. S51: In multiple time steps, the noise-free approximate image of the current time step is denoised according to the noise level of the previous time step to generate the pre-intermediate image of the previous time step.

[0182] The second computing unit is used to: execute step S52. S52: In multiple time steps, based on the increment matrix of the current time step, warp the pre-intermediate state image of the previous time step to obtain the intermediate state image of the previous time step.

[0183] In step S51, the intermediate state image of the previous time step t-1 Noise value at the current time step The noise-free approximation image at the current time step The following formula (16) applies between them.

[0184] Formula (16): .

[0185] For example, the previous time step of the preset time step T is T-1.

[0186] After obtaining the intermediate state image at a preset time step T. Then, the intermediate state image at the preset time step T. After complete denoising, a noise-free approximate image at a preset time step T can be obtained. Then, a noise-free approximation image at a preset time step T is obtained. A noise-free approximation image at a preset time step T is obtained based on the noise level at time step T-1. By adding noise, the pre-intermediate state image of the previous time step T-1 can be obtained. .

[0187] In step S52, for example, the pre-intermediate image at time step T-1 is compared with the incremental matrix at a preset time step T. Perform warping to obtain an intermediate state image at time step T-1. .

[0188] For example, in step S52, the intermediate state image of the previous time step can be obtained by the following formula (17).

[0189] Formula (17): .

[0190] in, This represents the result of the transformation of equation (16). This represents an image warping operation. During the training phase, this disclosure randomly samples time steps. It trains for a single time step for each batch, unlike the original iterative algorithm which trains multiple iterations.

[0191] Then, at time step T-1, the intermediate state image can be processed. Perform full denoising to obtain a noise-free approximate image at time step T-1. .

[0192] In steps S51 and S52, the pre-intermediate image can intuitively reflect the alignment effect of the current increment matrix. Based on the increment matrix of the current time step, the pre-intermediate image of the previous time step is distorted, and the geometric transformation is gradually accumulated to avoid image distortion and numerical instability caused by a large one-time transformation, thereby improving the stability of the registration process.

[0193] Table 1 shows the method for determining the registration homography of multimodal images.

[0194] In step 2, strategy settings, DDIM (denoising diffusion probabilistic models) generates a concise sampling sequence through deterministic sampling rules. It only takes 5-10 steps to complete denoising with the same effect as DDPM (denoising diffusion probabilistic model) which takes thousands of steps, thus improving inference speed.

[0195] This disclosure also provides a multimodal image registration method. This multimodal image registration method includes the multimodal image registration homography determination method provided in the above embodiments and the following step S6.

[0196] This disclosure also provides a multimodal image registration model, which includes the multimodal image registration homography determination model and registration execution subnet provided in some of the above embodiments.

[0197] The registration execution subnet is used to perform step S6. S6: Register the initial noisy image and the cross-modal reference image according to the registration homography matrix.

[0198] For example, in step S6, the registration homography matrix determined in steps S1-S4 is... (The globally optimal geometric transformation matrix) is applied to the initial noisy image. By registering the homography matrix Corresponding geometric transformations such as perspective, rotation, translation, and scaling are applied to the initial noisy image. The spatial coordinates are globally corrected to improve the transformed initial noisy image. With cross-modal reference image Precise alignment in spatial location reduces geometric differences between the two, resulting in a fully registered multimodal image pair. This improves the alignment accuracy of the multimodal images and enhances their noise robustness.

[0199] Furthermore, geometric transformation is performed only on the initial noisy image, while the cross-modal reference image remains fixed as the registration datum, avoiding secondary geometric deviations caused by simultaneous transformation of the two images and ensuring the uniqueness and stability of the registration datum.

[0200] In the registration process, the registration homography matrix used is... It is the globally optimal result obtained through multiple time steps of optimization, based on this registration homography matrix. The registration operation inherits the robustness of the entire process to noise and ensures registration accuracy, so that the registration results still have high reliability in noise degradation scenarios.

[0201] Figure 7 In the training method for the multimodal image registration homography determination network provided in this embodiment of the disclosure, the correspondence between the degree of distortion and the correlation map is shown. Figure 8 This is a flowchart illustrating the training method for a multimodal image registration homography determination network provided in this embodiment of the disclosure.

[0202] Please see Figure 7 and Figure 8 This disclosure also provides a training method for a registration homography determination network for multimodal images. The training method includes a first stage and a second stage.

[0203] In the first phase, the reconstruction items are used. and homography The loss function composed of the multimodal image denoising subnet and multimodal homography estimation subnet Optimization is performed. In the second stage, in the modal image denoising subnet... and multimodal homography estimation subnet In the case of parameters, use selection loss Independent training of adaptive homography selection subnets .

[0204] The first stage consists of multi-step training. In the i-th training step, an intermediate image is input into the multimodal image registration homography determination network. With cross-modal reference image To learn.

[0205] Among them, the intermediate state image input during the training process It can be made from the original clear image It is formed through distortion and noise addition. For example, for the original clear image. First, use truth labels. Perform distortion, then use the noise value. Add noise to obtain an intermediate image. .

[0206] In the first step of training, this disclosure uses a relatively large initial degree of distortion. To handle coarse alignment, where This indicates the upper limit of the distortion. In the second and third training steps, the degree of distortion is gradually reduced as the spatial deformation decreases. The degree of distortion for different training steps is defined as shown in the following formula (18).

[0207] Formula (18): .

[0208] For example, in the second training step, the degree of distortion .

[0209] In the third step of training, the degree of distortion .

[0210] To complement this coarse-to-fine distortion strategy, this disclosure further employs corresponding course learning on the spatial mismatch correlation map to achieve incremental matrix estimation from coarse to fine. Specifically, in the first step of training, the intermediate state image is used as the input to the network to determine the registration homography. With cross-modal reference image Then, a rough correlation diagram obtained from the deepest basic block is used. Incremental matrix estimation is performed. In the second and third training steps, more refined correlation maps are used respectively. And the most detailed correlation diagram Perform increasing matrix estimation. Utilize spatial mismatch correlation plots. , and The offset is obtained according to formula (13). Among them, truth labels It will depend on the degree of distortion Adjustments will be made. Homography loss during step training By penalizing the offset With truth labels The error between them is defined as, where homography loss is... It satisfies the following formula (19).

[0211] Formula (19): .

[0212] Here, represent Norm, The degree of distortion is represented as The truth value at time is the four-point offset. Among them, the offset depends on the degree of distortion. Adapting Truth Tags As Predicted supervisory labels, that is, in terms of the degree of distortion Truth labels in different situations different.

[0213] In addition to homography loss, this disclosure also employs reconstruction loss. To ensure noise-free prediction results Compared to the original clear image The fidelity between them, of which reconstruction loss It satisfies the following formula (20).

[0214] Formula (20): .

[0215] Here, reconstruction losses It remains unchanged across different training steps. Finally, the joint task network is in the... Total loss function in step training It satisfies the following formula (21).

[0216] Formula (21): .

[0217] in, and These are the weight parameters for different loss terms.

[0218] This disclosure is obtained through step i. Denoising subnet for multimodal images and multimodal homography estimation subnet Optimize.

[0219] During the second stage of training the AHS subnetwork, the multimodal image denoising subnetwork... and multimodal homography estimation subnet The parameters are frozen, meaning the multimodal image denoising subnet... and multimodal homography estimation subnet The parameters do not participate in any gradient updates or undergo any parameter value changes during the training of the AHS subnet.

[0220] In the second stage, this disclosure introduces a selection loss to train the AHS subnetwork, aiming to minimize the difference between the predicted registration error and the ground truth registration error. Here, the ground truth registration error is defined as the homography loss of the joint task network during the first training step. This step corresponds to the highest degree of distortion. This helps the AHS subnetwork maintain robustness under varying degrees of deformation. Choosing the loss function... It satisfies the following formula (22).

[0221] Formula (22): .

[0222] Here, This indicates the prediction error. This represents the true registration error, which is the error recorded at time step in the first training step. Calculated homography loss The normalized form. By... Normalization The AHS subnetwork can achieve stable error prediction and exhibit consistency across different samples.

[0223] This disclosure, by selecting a loss function Optimize the AHS subnet.

[0224] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0225] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for determining the homography of multimodal images, characterized in that, include: The initial noisy image is embedded into the diffusion denoising process to generate an initial intermediate image; Guided by a cross-modal reference image, the initial intermediate image is progressively denoised based on a diffusion denoising process to generate intermediate images of multiple intermediate steps, and the intermediate image of the current time step is completely denoised to generate a noise-free approximate image of the current time step. In multiple time steps, the increment matrix for the current time step is predicted based on the cross-modal reference image and the noise-free approximation image for the current time step. Among the incremental matrices of the multiple time steps, the incremental matrix with the smallest registration error is selected as the registration incremental matrix, and the homography matrix of the time step corresponding to the registration incremental matrix is ​​determined as the registration homography matrix.

2. The method for determining the registration homography of multimodal images according to claim 1, characterized in that, The step of progressively denoising the initial intermediate image based on a diffusion denoising process, guided by a cross-modal reference image, to generate intermediate images at multiple intermediate steps, and performing complete denoising on the intermediate image at the current time step to generate a noise-free approximate image at the current time step, includes: At multiple time steps, guided by the cross-modal reference image and combined with the noise level of the current time step, the noise value of the intermediate image at the current time step is estimated; Based on the noise value of the intermediate image at the current time step, the intermediate image at the current time step is completely denoised to obtain the noise-free approximate image at the current time step.

3. The method for determining the registration homography of multimodal images according to claim 2, characterized in that, The step of estimating the noise value of the intermediate image at the current time step, guided by the cross-modal reference image and in conjunction with the noise level at the current time step, includes: The feature information of the cross-modal reference image is extracted and combined with the noise level of the current time step to generate a cross-modal guidance term; Hierarchical features are extracted from the intermediate state image at the current time step, and the prediction noise at the current time step is predicted under the guidance of the cross-modal guidance term.

4. The method for determining the registration homography of multimodal images according to claim 2 or 3, characterized in that, Also includes: At multiple time steps, the noise-free approximate image at the current time step is noise-added according to the noise level of the previous time step to generate a pre-intermediate image of the previous time step. In multiple time steps, the pre-intermediate image of the previous time step is warped based on the incremental matrix of the current time step to obtain the intermediate image of the previous time step.

5. The method for determining the registration homography of multimodal images according to claim 1, characterized in that, The step of embedding the initial noisy image into the diffusion denoising process to generate an initial intermediate image includes: Analyze the initial noise level of the initial noise image, and determine the time step with the smallest absolute difference from the initial noise level during the diffusion denoising process as the preset time step; The initial noisy image is embedded into the preset time step of the diffusion denoising process, and an intermediate image of the preset time step is generated based on the initial noisy image.

6. The method for determining the registration homography of multimodal images according to claim 5, characterized in that, In the analysis of the initial noise level of the initial noise image, the time step with the smallest absolute difference from the initial noise level during the diffusion denoising process is determined as a preset time step, wherein the preset time step satisfies: ； Where T is the preset time step. The initial noise level, For noise scheduling during the diffusion process; among which, satisfy: ,in, This is the hyperparameter for the current time step.

7. The method for determining the registration homography of multimodal images according to claim 1, characterized in that, The step of predicting the increment matrix for the current time step based on the cross-modal reference image and the noise-free approximation image for the current time step across multiple time steps includes: Multiple first-scale feature maps are extracted from the noise-free approximate image at the current time step to form a first multi-scale feature set, and multiple second-scale feature maps are extracted from the cross-modal reference image to form a second multi-scale feature set. Based on the first multi-scale feature set and the second multi-scale feature set, the offset between the noise-free approximate image and the cross-modal reference image at the current time step is determined, and the increment matrix at the current time step is determined based on the offset.

8. The method for determining the registration homography of multimodal images according to claim 7, characterized in that, The step of determining the offset between the noise-free approximate image and the cross-modal reference image at the current time step based on the first multi-scale feature set and the second multi-scale feature set, and determining the increment matrix at the current time step based on the offset, includes: Based on the first multi-scale feature set and the second multi-scale feature set, determine the cosine similarity at multiple scales between the first position in the noise-free approximate image and the second position in the cross-modal reference image. Upsampling is performed on the cosine similarity at multiple scales to obtain a spatial mismatch correlation map between the first position and the second position at multiple scales. The spatial mismatch correlation graph at multiple scales is decoded into the offset at the current time step; Based on the direct linear transformation algorithm, the increment matrix of the current time step is calculated according to the offset of the current time step.

9. The method for determining the registration homography of multimodal images according to claim 8, characterized in that, The second position is located within the neighborhood of a square window centered on the first position and with a radius of a first preset radius.

10. The method for determining the registration homography of multimodal images according to claim 8, characterized in that, In determining the cosine similarity at multiple scales between a first position in the noise-free approximate image and a second position in the cross-modal reference image based on the first multi-scale feature set and the second multi-scale feature set, the cosine similarity satisfies: ； in, Let represent the cosine similarity between the first position p in the noise-free approximation image and the second position u in the cross-modal reference image at the nth scale. This represents the feature vector at position p on the feature map of the noise-free approximation image at the first position. This represents the feature vector at the second position u on the nth scale feature map of the cross-modal reference image; By upsampling the cosine similarity at multiple scales to obtain a spatial mismatch correlation map between the first position and the second position at multiple scales, the spatial mismatch correlation map satisfies: ； in, This is the spatial mismatch correlation diagram; In decoding the spatial mismatch correlation map at multiple scales into the offset at the current time step, the offset satisfies: ； in, This represents the offset at time step t; In the calculation of the increment matrix based on the direct linear transformation algorithm and the offset, the increment matrix satisfies: ； in, This is the increment matrix.

11. The method for determining the registration homography of multimodal images according to claim 1, characterized in that, The step of selecting the incremental matrix with the smallest registration error from the incremental matrices of the multiple time steps as the registration incremental matrix, and determining the homography matrix of the time step corresponding to the registration incremental matrix as the registration homography matrix, includes: Obtain the first depth feature map of the current time step from the noise-free approximate image of the current time step; Obtain a second depth feature map from the cross-modal reference image; The coordinate stream for the current time step is generated using coordinate stream encoding based on the increment matrix of the current time step; Using the coordinate flow of the current time step, the first depth feature map of the current time step is warped to obtain the first aligned version map of the current time step; Estimate the registration error between the first aligned version map at the current time step and the second depth feature map at the current time step, and use it as the registration error of the incremental matrix at the current time step; Among the incremental matrices of the multiple time steps, the incremental matrix with the smallest registration error is selected as the registration incremental matrix; The homography matrix of the time step corresponding to the registration increment matrix is ​​determined as the registration homography matrix.

12. The method for determining the registration homography of multimodal images according to claim 11, characterized in that, In determining the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix, the homography matrix satisfies: ； in, Let be the homography matrix at the current time step t; The homography matrix for the next time step t+1 .

13. A registration method for multimodal images, characterized in that, include: The method for determining the registration homography of multimodal images as described in any one of claims 1-12; Based on the registration homography matrix, the initial noisy image and the cross-modal reference image are registered.

14. A registration homography determination network for multimodal images, characterized in that, include: An embedding subnet is used to: embed the initial noisy image into the diffusion denoising process to generate an initial intermediate image; A multimodal image denoising subnet is used to: perform stepwise denoising on the initial intermediate image based on a diffusion denoising process, guided by a cross-modal reference image, to generate intermediate images of multiple intermediate steps, and to perform complete denoising on the intermediate image at the current time step to generate a noise-free approximate image at the current time step. A multimodal homography estimation subnet is used to: predict the increment matrix and homography matrix of the current time step based on the cross-modal reference image and the noise-free approximation image of the current time step at multiple time steps; An adaptive homography selection subnet is used to: select the incremental matrix with the smallest registration error among the incremental matrices of multiple time steps as the registration incremental matrix, and determine the homography matrix of the time step corresponding to the registration incremental matrix as the registration homography matrix.

15. The multimodal image registration homography determination network according to claim 14, characterized in that, The multimodal image denoising subnetwork includes: a noise estimation unit and a complete denoising unit; The noise estimation unit is used to: estimate the noise value of the intermediate image at the current time step, guided by the cross-modal reference image and in combination with the noise level at the current time step, at multiple time steps. The complete denoising unit is used to: perform complete denoising on the intermediate image at the current time step based on the noise value of the intermediate image at the current time step, so as to obtain the noise-free approximate image at the current time step.

16. The multimodal image registration homography determination network according to claim 14, characterized in that, The noise estimation unit includes a cross-modal guidance module and a denoising U-Net module; The cross-modal guidance module is used to: extract feature information from the cross-modal reference image and generate a cross-modal guidance term in combination with the noise level of the current time step; The denoising U-Net module is used to: extract hierarchical features from the intermediate image at the current time step, and predict the prediction noise at the current time step under the guidance of the cross-modal guidance term.

17. The multimodal image registration homography determination network according to claim 15 or 16, characterized in that, An iterative calculation unit, comprising: a first calculation unit and a second calculation unit; The first computing unit is used to: add noise to the noise-free approximate image at the current time step according to the noise level of the previous time step, so as to generate the pre-intermediate image of the previous time step. The second computing unit is used to: warp the pre-intermediate image of the previous time step based on the incremental matrix of the current time step, so as to obtain the intermediate image of the previous time step.

18. The multimodal image registration homography determination network according to claim 14, characterized in that, The embedded subnet includes: a third computing unit and an embedding unit; The third calculation unit is used to: analyze the initial noise level of the initial noise image, and determine the time step with the smallest absolute difference from the initial noise level during the diffusion denoising process as the preset time step; The embedding unit is used to: embed the initial noise image into the preset time step of the diffusion denoising process, and generate an intermediate image of the preset time step based on the initial noise image.

19. The multimodal image registration homography determination network according to claim 14, characterized in that, The multimodal homography estimation subnetwork includes: a multi-scale feature extraction unit and a cross-modal homography estimation unit; The multi-scale feature extraction unit is used to: extract multiple first-scale feature maps from the noise-free approximate image at the current time step to form a first multi-scale feature set, and extract multiple second-scale feature maps from the cross-modal reference image to form a second multi-scale feature set. The cross-modal homography estimation unit is used to: determine the offset between the noise-free approximate image and the cross-modal reference image at the current time step based on the first multi-scale feature set and the second multi-scale feature set, and determine the increment matrix at the current time step based on the offset.

20. The multimodal image registration homography determination network according to claim 19, characterized in that, The cross-modal homography estimation unit includes a correlation calculation module, an upsampling module, a decoding module, and an incremental matrix estimation module; The correlation calculation module is used to: determine the cosine similarity at multiple scales between the first position in the noise-free approximate image and the second position in the cross-modal reference image, based on the first multi-scale feature set and the second multi-scale feature set; The upsampling module is used to: upsample the cosine similarity at multiple scales to obtain a spatial mismatch correlation map between the first position and the second position at multiple scales; The decoding module is used to: decode the spatial mismatch correlation map at multiple scales into the offset of the current time step; The incremental matrix estimation module is used to: calculate the incremental matrix of the current time step based on the offset of the current time step using the direct linear transformation algorithm.

21. The multimodal image registration homography determination network according to claim 14, characterized in that, The adaptive homography selection subnet includes: a first feature extraction unit, a second feature extraction unit, a coordinate flow generation unit, an alignment unit, an error estimation unit, a registration increment matrix determination unit, and a homography matrix determination unit; The first feature extraction unit is used to: obtain a first depth feature map of the current time step from the noise-free approximate image of the current time step; The second feature extraction unit is used to: obtain a second depth feature map from the cross-modal reference image; The coordinate stream generation unit is used to: generate a coordinate stream for the current time step based on the increment matrix of the current time step using coordinate stream encoding; The alignment unit is used to: warp the first depth feature map of the current time step using the coordinate flow of the current time step to obtain a first aligned version map of the current time step; The error estimation unit is used to: estimate the registration error between the first aligned version map at the current time step and the second depth feature map at the current time step, as the registration error of the incremental matrix at the current time step; The registration increment matrix determination unit is used to: select the increment matrix with the smallest registration error from the increment matrices of the multiple time steps, and use it as the registration increment matrix; The homography matrix determination unit is used to: determine the homography matrix of the time step corresponding to the registration increment matrix as the registration homography matrix.

22. A multimodal image registration model, characterized in that, include: The registration homography determination network for multimodal images as described in any one of claims 14-21; A registration execution subnet is used to: register the initial noisy image and the cross-modal reference image according to the registration homography matrix.

Images