Characterization decoupling-based multi-modal image registration method and system

Abstract

The invention relates to a medical image processing technology, and discloses a multi-modal image registration method and system based on representation decoupling. Respectively coding a moving image and a fixed image through a dual-branch multi-scale coding network with independent parameters, and extracting modal-independent features through a multilayer modal-independent contrast loss constraint in training; the features of different scales are input into cascaded multi-scale feature registration modules, registration is carried out step by step according to the scales from small to large, and each module receives the deformation field and the deformation feature of the previous scale, outputs the deformation field and the deformation feature of the current scale and transmits the deformation field and the deformation feature backwards; finally, the deformation field acts on the moving image to obtain a registration result. According to the multi-modal medical image registration method and device, modal irrelevant feature extraction is achieved through double-branch independent coding and multi-layer comparison loss constraint cooperation, error accumulation is relieved through deformation field and deformation feature joint transmission, and the precision and stability of multi-modal medical image registration are remarkably improved.

Description

Technical Field

[0001] This application relates to medical image processing technology, specifically to medical image registration technology, and particularly to a multimodal medical image registration method based on deep learning. Background Technology

[0002] In medical imaging diagnosis and treatment planning, multimodal medical image registration is a fundamental and crucial technical task. Different imaging modalities can provide different information about the human body; for example, CT images can clearly show the spatial structure of high-density tissues such as bones, while MR images can provide rich details of soft tissues. By spatially aligning these different modalities of medical images, the complementary information of each modality can be comprehensively utilized, providing clinicians with a more comprehensive diagnostic basis.

[0003] In surgical navigation scenarios, high-resolution preoperative MR images provide precise anatomical information, while real-time intraoperative CT images display the current organ location and spatial relationships of surgical instruments. Precise registration of preoperative MR images with intraoperative CT images helps surgeons accurately locate lesions, plan puncture paths, and avoid critical structures such as important blood vessels and nerves during surgery. In radiotherapy planning scenarios, multimodal images acquired at different times need to be registered to assess tumor growth and displacement of surrounding tissues. In multicenter medical research scenarios, multimodal images from different medical institutions often exhibit spatial inconsistencies due to differences in imaging equipment, patient positioning, and acquisition times. This makes direct comparison of the characteristics of the same anatomical structure difficult, necessitating precise multimodal registration techniques to establish spatial correspondences.

[0004] In existing technologies, traditional multimodal image registration methods are mainly based on iterative optimization techniques. They achieve image alignment by defining a similarity metric function and iteratively solving the deformation field. While these methods offer some stability in registration accuracy, they are computationally expensive, often requiring several minutes or even longer for a single registration, making them unsuitable for real-time clinical applications. Furthermore, these methods are prone to getting trapped in local optima, especially when there are large initial displacements or complex deformations between images, resulting in poor robustness of the registration results. In recent years, with the development of deep learning technology, unsupervised or weakly supervised multimodal registration methods based on deep learning have gradually become a research hotspot. These methods learn the registration model offline on large datasets, requiring only a single forward computation to obtain the deformation field during online inference. The registration time for a single operation can be reduced to the second level, significantly improving registration efficiency.

[0005] However, existing deep learning-based multimodal registration techniques still have many limitations in practical applications. The first common multimodal registration method is based on the UNet architecture, which stitches multimodal images along the channel dimension and inputs them into the network. The model directly evaluates the structural alignment between the two images and outputs the deformation field. The main problem with this type of method is that, due to the significant intensity differences between different modal images, the network struggles to effectively learn registration features reflecting anatomical correspondences from image groups with large intensity distribution differences, resulting in limited registration accuracy. The second method is based on generative models, which transform multimodal registration into a monomodal registration problem by generating images of one modality from images of another. While this type of method alleviates the modal difference problem to some extent, the registration accuracy heavily depends on the performance of the generative network. Current image generation models often produce poor results and artifacts for small anatomical structures in medical images, and these generation errors directly affect the subsequent registration quality. The third method uses a pyramid registration architecture, which transfers the deformation field step by step from a low-resolution image to a high-resolution image. However, when dealing with large deformation scenes, the registration error generated at the early scale will accumulate and amplify as the deformation field is transferred step by step, eventually affecting the overall registration effect.

[0006] Therefore, there is an urgent need for a new multimodal image registration technology that can effectively address the dual challenges of large modal intensity differences and large deformation amplitudes, and achieve accurate, robust, and efficient multimodal medical image registration. Summary of the Invention

[0007] The purpose of this application is to provide a multimodal image registration method and system based on representation decoupling to solve the problems mentioned in the background art.

[0008] This application discloses a multimodal image registration method based on representation decoupling, including the following steps:

[0009] Step S1: Obtain the moving image and the fixed image to be registered, and preprocess the moving image and the fixed image;

[0010] Step S2: Input the moving image and the fixed image into the corresponding branches of the dual-branch multi-scale coding network for encoding to obtain a series of moving image features and fixed image features of different scales from small to large;

[0011] The dual-branch multi-scale coding network comprises two branches with identical structures but independent parameters, and is constrained by a multi-layer modality-independent contrastive loss during training to extract modality-independent features.

[0012] Step S3: Input the moving image features and fixed image features of different scales into a registration network containing multiple cascaded multi-scale feature registration modules, and perform registration step by step in order of scale from smallest to largest;

[0013] In the first scale, the initial deformation field and deformation features are calculated based on the moving image features and fixed image features at the smallest scale. In subsequent scales, the input of each multi-scale feature registration module includes: the moving image features at the current scale, the fixed image features at the current scale, and the deformation field and deformation features output from the previous scale. Based on the input, the multi-scale feature registration module outputs the deformation field and deformation features at the current scale and passes them to the multi-scale feature registration module at the next scale.

[0014] Step S4: Apply the deformation field output from the last scale to the moving image to obtain the registered image.

[0015] In a preferred embodiment, the bi-branch multi-scale coding network and the registration network are pre-trained as follows: based on moving and fixed images in the training set, features at different scales are obtained through the bi-branch multi-scale coding network, and a multi-modal independent contrast loss is calculated, wherein different modal image features of the same subject are regarded as positive class pairs, and different modal image features of different subjects are regarded as negative class pairs; the multi-modal independent contrast loss is combined with similarity loss and deformation field smoothness loss to form an overall loss, and the parameters of the bi-branch multi-scale coding network and the registration network are updated using the overall loss.

[0016] In a preferred embodiment, the multimodal independent contrast loss is calculated using the following formula:

[0017]

[0018] Where l represents the number of layers in the coding network, w l The weights represent the loss of the l-th layer. Represents moving image features based on layer l and fixed image features The calculated contrast loss.

[0019] In a preferred embodiment, the multi-scale feature registration module includes: an input adjustment unit, configured to receive the deformation field and deformation features output from the previous scale, and upsample and adjust the deformation features to a scale that matches the current scale features to obtain reference deformation features; a feature deformation unit, configured to perform spatial transformation (Warp) on the moving image features at the current scale based on the deformation field output from the previous scale or the scale-adjusted deformation field to obtain deformed moving image features at the current scale; and a feature fusion inference unit, configured to combine the deformed moving image features at the current scale, the fixed image features at the current scale, and the reference deformation features, and process them through a convolutional neural network to output the deformation field and deformation features at the current scale.

[0020] In a preferred embodiment, during the reasoning phase, step S3 further includes an adaptive recursive reasoning process:

[0021] At the current registration scale, the anatomical structure labels of the moving image are transformed using the currently calculated deformation field, and the Dice coefficient between the transformed labels and the labels of the fixed image is calculated.

[0022] Compare the current Dice coefficient with the reference Dice coefficient of the current record;

[0023] When the Dice coefficients are improved compared to the currently recorded reference Dice coefficients, the currently calculated deformation field is used as the prior input, and the multi-scale feature registration module is recursively called to perform inference again at this scale and update the reference Dice coefficients.

[0024] When the Dice coefficient no longer increases or reaches the preset stopping condition, the recursion stops, and the optimal deformation field and deformation characteristics obtained at the current scale are passed to the next scale.

[0025] In a preferred embodiment, the image preprocessing in step S1 includes:

[0026] For CT images, adjust the window width and window level, and adjust the Hounsfield unit (HU) range to [-200, 400];

[0027] For MR images, the 99.99th percentile is taken as the maximum intensity value of the image for intensity range mapping, and N4 intensity range correction is performed.

[0028] The moving and stationary images are resampled to adjust the spatial resolution of the images to 1.75×1.75×2mm;

[0029] The resampled image is center-cropped to reduce the size of the input image to 160×160×64 pixels;

[0030] Normalize all image pixel values ​​to the interval [0,1].

[0031] In a preferred embodiment, each branch of the dual-branch multi-scale coding network consists of three sub-modules connected in series;

[0032] Each of the sub-modules is implemented using a two-layer convolutional neural network with residual connections;

[0033] The output feature size of each submodule is half the input feature size of that submodule.

[0034] In a preferred embodiment, the multi-scale feature registration module is performed through the following specific operations:

[0035] The deformation characteristics of the input adjustment unit output at the previous scale Perform upsampling convolution operations to output the reference deformation features at the current scale.

[0036] The deformation field φ output by the characteristic deformation unit at the previous scale i+1 Upsampling was performed to obtain and utilize Features of moving images at the current scale Perform a spatial transformation operation (Warp) to obtain the deformed and moved image features at the current scale.

[0037] The feature fusion inference unit will use fixed image features at the current scale. The deformed and moving image features and the reference deformation features Channel stitching is performed, and the stitched features are processed through convolutional layers and a registration head network (RegHead) to output the deformation features at the current scale. and deformation field φ i .

[0038] In a preferred embodiment, data augmentation is also included before training based on the training set images:

[0039] Random rotation, scaling changes, and elastic deformation were applied to the MR images in the training set;

[0040] The same data augmentation strategy was used for CT images in the training set as for MR images to maintain consistency of input conditions.

[0041] In a preferred embodiment, the contrast loss of the l-th layer Calculated using the following formula:

[0042]

[0043] Among them, (v,v) + () represents a positive class image pair, i.e., different modal image features of the same subject; represents negative class image pairs, i.e., different modal image features of different subjects; N represents the number of negative class samples; τ represents the temperature parameter used to control the distance between different categories; · represents the dot product operation.

[0044] In a preferred embodiment, the similarity loss employs a loss function based on the local neighborhood self-similarity descriptor (MIND), the calculation steps of which include: for each voxel x in the image, calculating the difference D between it and its neighboring voxels. p :D p (I,x,α)=C·(I(x)-I(x+α)) 2 Where I represents the image, x represents the voxel position, α represents the neighborhood offset, and C is a constant; the local variance V(I,x) is calculated as a normalization factor: in, Represent the neighborhood; and construct the MIND vector. Where Z is a normalization constant; for the deformed moving image and fixed image I fixed Calculate the L1 distance between all voxels as the similarity loss. Where, N voc This indicates the total number of voxels.

[0045] In a preferred embodiment, the deformation field smoothness loss Calculated using the following formula:

[0046]

[0047] Where, φ d (x) represents the d-th directional component of the deformation field at position x. This represents the gradient operator, which is used to constrain the deformation field to produce smooth deformation.

[0048] In a preferred embodiment, the total loss L is calculated using the following formula:

[0049]

[0050] Among them, I f Indicates a fixed image, I m Represents a moving image, φ represents the deformation field, and F f and F m λ1 and λ2 represent fixed image features and moving image features, respectively, and are weighting coefficients.

[0051] This application also discloses a multimodal image registration system based on representation decoupling, including:

[0052] The preprocessing module is used to acquire the moving image and the fixed image to be registered, and to preprocess the moving image and the fixed image.

[0053] A dual-branch multi-scale coding module is used to input the moving image and the fixed image into the corresponding branches of the dual-branch multi-scale coding network for encoding, thereby obtaining a series of moving image features and fixed image features at different scales from small to large. The dual-branch multi-scale coding network contains two branches with the same structure but independent parameters, and is constrained by a multi-layer modality-independent contrastive loss during training to extract modality-independent features.

[0054] The registration module is used to input the moving image features and fixed image features of different scales into a registration network containing multiple cascaded multi-scale feature registration modules, and perform registration step by step in order of scale from smallest to largest. In the first scale, the initial deformation field and deformation features are calculated based on the moving image features and fixed image features of the smallest scale. In subsequent scales, the input of each multi-scale feature registration module includes: the moving image features of the current scale, the fixed image features of the current scale, and the deformation field and deformation features output from the previous scale. Based on the input, the multi-scale feature registration module outputs the deformation field and deformation features of the current scale and passes them to the multi-scale feature registration module of the next scale.

[0055] The deformation application module is used to apply the deformation field output from the last scale to the moving image to obtain the registered image.

[0056] To address the inherent technical challenges of large modal intensity differences and large deformation amplitudes in multimodal medical image registration tasks, this application achieves significant technical improvements through a series of interrelated technical means.

[0057] First, in the data preprocessing stage, window width and level adjustments and quantile normalization were applied to both CT and MR images, along with N4 intensity range correction. This effectively eliminated inconsistencies in intensity distribution caused by different modalities and acquisition devices, creating uniform input conditions for subsequent modality-independent feature extraction. Furthermore, spatial resolution was unified through resampling, and the region of interest was focused on via center cropping, reducing computational costs while preserving crucial anatomical information. During the training phase, the same data augmentation strategy was used for both MR and CT images, ensuring the model's robustness to spatial transformations and avoiding disruption of intermodal spatial correspondences due to inconsistent augmentation strategies.

[0058] In the feature encoding stage, the dual-branch multi-scale encoding network features two branches with identical structures but completely independent parameters. This allows each branch to specialize in learning specific modalities, avoiding feature confusion and optimization conflicts caused by simply stitching together images of different modalities in traditional methods. Each branch consists of multiple sub-modules connected in series with residual connections. These residual connections not only promote efficient gradient propagation but also enable hierarchical reuse of features, allowing the encoder to simultaneously capture both coarse-scale global structure and fine-scale local details. More importantly, by applying multi-level modality-independent contrastive loss constraints across multiple layers of the encoding network, the model is forced to learn to bring different modalities of the same subject closer together in the feature space, while pushing features from different subjects further apart. The profound effect of this contrastive learning mechanism lies in the fact that it not only constrains the modality independence of features at a single level of abstraction but also systematically guides the direction of feature learning at different levels of abstraction through multi-level weighted losses. This ensures that each representation level, from low-level texture to high-level semantics, focuses on modality-invariant information dimensions such as anatomical structure, while actively suppressing dependence on modality-specific information such as intensity distribution. This multi-level feature decoupling effect significantly improves the model's robustness to modal differences. Even when faced with modal combinations or imaging parameter changes that were not seen during training, the model can still maintain stable registration performance.

[0059] In the registration stage, the multi-scale feature registration module employs a pyramid architecture, achieving progressive alignment from coarse to fine through hierarchical registration from the smallest to the largest scale. At each scale, the multi-scale feature registration module receives not only the deformation field from the previous scale but also the deformation features from the previous scale. This dual information transmission mechanism produces technical effects that surpass the accumulation of simple geometric transformations. Specifically, after the input adjustment unit upsamples and convolves the deformation features from the previous scale, the resulting reference deformation features carry registration strategy information extracted during the previous scale registration process, such as implicit knowledge about which regions require greater deformation and which regions have more reliable feature matching. The feature deformation unit uses the upsampled deformation field to perform spatial transformation on the moving image features, achieving preliminary alignment at the feature level. The feature fusion inference unit concatenates the fixed image features, the deformed moving image features, and the reference deformation features, enabling the convolutional network to simultaneously perceive the target state, the current alignment state, and the registration experience from the previous scale in a unified representation space, thereby making more informed deformation field predictions. The synergistic effect of this information fusion is that the registration at the current scale is no longer a blind feature matching, but an intelligent refinement based on making full use of coarse-scale registration experience, which effectively avoids the problem of registration error accumulating and amplifying with scale transmission in traditional methods.

[0060] For similarity measurement, the MIND loss function based on local neighborhood self-similarity descriptors is adopted. Compared with traditional intensity difference measures, it can capture the local structural patterns of the image rather than absolute intensity values, making this loss function inherently robust to intensity distribution differences in multimodal images. The MIND loss constructs a structural descriptor that is insensitive to changes in imaging parameters by calculating the intensity difference between each voxel and its neighboring voxels and normalizing it with local variance. The deformation field smoothness loss ensures the continuity and physical rationality of deformation by penalizing the spatial gradient of the deformation field, avoiding unnatural folds or breaks in the deformation field. The overall loss function is a weighted combination of multimodal independent contrast loss, similarity loss, and smoothness loss, achieving a multi-objective optimization balance of feature learning, registration accuracy, and deformation smoothness.

[0061] During the inference phase, the adaptive recursive inference strategy introduces a dynamic optimization mechanism based on registration quality feedback. At each registration scale, by applying the current deformation field to the anatomical structure label of the moving image and calculating the Dice coefficient between the deformed label and the fixed image label, the model can quantitatively evaluate the registration quality at the current scale. When the Dice coefficient improves compared to the reference value, it indicates that the current registration direction is correct. At this point, the current deformation field is used as a priori input to recursively call the registration module, which can further optimize the registration effect within the same scale. The technical effect of this intra-scale recursive mechanism is that it provides multiple opportunities for iterative optimization at each scale, so that the registration process is no longer limited by the network's single forward inference capability, but gradually approaches the optimal registration state at that scale through iteration. When the Dice coefficient no longer improves, it indicates that a local optimum has been reached at the current scale. At this point, recursion stops and the optimal result is passed to the next scale. This adaptive stopping criterion allows different cases and different anatomical regions to automatically determine the number of iterations based on their registration difficulty, avoiding computational waste caused by excessive iteration and insufficient accuracy caused by underoptimization. From a system perspective, the adaptive recursive inference strategy combines the inter-scale transfer of pyramid registration with intra-scale optimization based on quality feedback, forming a two-level iterative registration framework that effectively alleviates the inherent defect of registration error accumulating step by step in traditional methods.

[0062] By combining the synergistic effects of the aforementioned techniques, this application achieves the following overall technical effects: At the feature extraction level, through the synergistic effect of dual-branch parameter decoupling and multi-layer contrast constraints, the extracted feature representations possess high modality independence and discriminability, enabling accurate capture of anatomical correspondences across intensity differences in different modalities; at the registration optimization level, through the joint transfer of deformation fields and deformation features, and the progressive refinement of the pyramid architecture, the registration process achieves effective information accumulation and error suppression from coarse to fine; at the inference adaptation level, through the quality feedback mechanism introduced by the adaptive recursive inference strategy, the registration process possesses the ability to dynamically optimize for specific cases, achieving excellent average performance not only at the overall dataset level but also at the individual case level, realizing high-precision personalized registration. The synergistic effect of these techniques allows this application to significantly improve the accuracy, robustness, and clinical applicability of multimodal medical image registration while maintaining good deformation field smoothness, providing more reliable technical support for applications such as medical image analysis, surgical navigation, and radiotherapy planning.

[0063] The specification of this application contains numerous technical features distributed across various technical solutions. Listing all possible combinations of these technical features (i.e., technical solutions) would make the specification excessively lengthy. To avoid this problem, the various technical features disclosed in the above-described invention, the various technical features disclosed in the following embodiments and examples, and the various technical features disclosed in the accompanying drawings can be freely combined to form various new technical solutions (all of which are considered to have been described in this specification), unless such a combination of technical features is technically infeasible. For example, one example discloses feature A+B+C, and another example discloses feature A+B+D+E. Features C and D are equivalent technical means that serve the same function, and technically only one needs to be used; they cannot be used simultaneously. Feature E can technically be combined with feature C. Therefore, the solution A+B+C+D should not be considered as described because it is technically infeasible, while the solution A+B+C+E should be considered as described. Attached Figure Description

[0064] Figure 1 This is an overall framework diagram of a multimodal image registration method according to an embodiment of this application.

[0065] Figure 2 This is a schematic diagram illustrating the principle of multi-modal independent contrast loss according to an embodiment of this application.

[0066] Figure 3 This is a schematic diagram of the structure of a multi-scale feature registration module according to an embodiment of this application.

[0067] Figure 4This is a flowchart illustrating a multimodal image registration method based on representation decoupling according to an embodiment of this application.

[0068] Figure 5 This is a schematic diagram of the structure of a multimodal image registration system based on representation decoupling according to an embodiment of this application. Detailed Implementation

[0069] In the following description, many technical details are presented to help the reader better understand this application. However, those skilled in the art will understand that the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments.

[0070] Explanation of some concepts:

[0071] Moving image: refers to an image that requires spatial morphological changes (deformation) during registration to align with a reference coordinate system. In the embodiments of this application, it typically refers to the source image to be registered, such as a magnetic resonance (MR) image.

[0072] Fixed image: refers to an image that serves as a spatial reference during registration and does not undergo morphological changes. In the embodiments of this application, it typically refers to the target image, such as a computed tomography (CT) image.

[0073] Representation decoupling refers to the process of separating information in a multimodal image into modality-related information (such as intensity distribution and imaging style) and modality-independent information (such as anatomical structure and spatial topology). This application achieves the extraction of modality-independent features through bi-branch independent parametric encoding and multi-layer modality-independent contrastive loss constraints.

[0074] Modal-independent features are feature representations extracted from images of different modalities that reflect common information such as anatomical structure and texture, without depending on the intensity distribution of a specific modality. These features have cross-modal consistency and are key to achieving accurate multimodal registration.

[0075] Multi-Layer Modal-Independent Contrastive Loss (MCL): This refers to a loss function mechanism proposed in this application for training dual-branch encoding networks. This mechanism simultaneously applies contrast constraints at multiple layers of the encoding network, bringing together image features from different modalities of the same subject (positive class pairs) and distancing image features from different subjects (negative class pairs), thereby constraining the network to learn cross-modal consistent anatomical representations.

[0076] Multi-Scale Feature Registration (MSFR) module: This refers to the core functional unit in the registration network of this application, which is used to perform registration inference at a specific resolution scale. This module receives the deformation field and deformation features of the previous scale and the image features of the current scale, and outputs the deformation field and deformation features of the current scale.

[0077] Deformation field (φ): A vector field describing the spatial displacement of each pixel or voxel in an image. For three-dimensional medical images, the deformation field contains three directional components at each voxel location, representing displacement in the x, y, and z directions, respectively. Applying the deformation field to a moving image allows for spatial transformation of the image.

[0078] Deformation Feature: In the multi-scale registration process, this refers to the implicit feature representation extracted by the registration module and passed between scales, containing registration context information and intermediate registration states. It differs from the explicit deformation field, carries prior knowledge from the registration process, and is used to guide the registration calculation at the next scale.

[0079] Adaptive Recursive Inference: This refers to a dynamic optimization strategy employed during the inference phase. Within each registration scale, a registration quality index (such as the Dice coefficient) is used as feedback to recursively call the registration module to iteratively optimize the deformation field until the registration quality index no longer improves or reaches a preset stopping condition. This achieves the optimal registration effect at that scale before being passed to the next scale.

[0080] MIND (Modality Independent Neighborhood Descriptor): This refers to an image descriptor based on local neighborhood self-similarity. It describes local structural patterns by calculating the intensity difference distribution between a pixel and its neighboring pixels. It is robust to nonlinear intensity changes between modalities and is often used to calculate the similarity loss of multimodal images.

[0081] Spatial transformation (Warp): refers to the operation of resampling an image based on a deformation field. By using interpolation methods, each pixel or voxel in the image is repositioned according to the displacement indicated by the deformation field, thereby achieving image deformation.

[0082] The following is a brief summary of some of the innovative aspects of this application:

[0083] In summary, this application addresses the long-standing technical bottlenecks in multimodal medical image registration, namely, large modal intensity differences and significant deformation amplitudes. It proposes a registration method based on the coupling of representation decoupling and adaptive recursive reasoning. The innovative aspect of this method lies in its use of a dual-branch multi-scale coding network (see...). Figure 1 The structural arrangement of maintaining parameter independence in branches A and B, combined with multilevel modal independent contrast loss, is a key feature. (see Figure 2 MCLoss in the encoder output features at each level and The positive and negative class pair constraints applied systematically suppress modality-related intensity pattern information in the feature space, while selectively enhancing modality-independent information such as anatomical structures. This synergistic mechanism of bi-branch parameter decoupling and multi-layer contrastive constraints is not simply a superposition of two known techniques, but rather creates the necessary feature differentiation conditions for contrastive learning through parameter independence. At the same time, the multi-layer application of contrastive loss inversely constrains the optimization direction of each branch parameter, forming a mutually dependent and interdependent technical relationship.

[0084] Furthermore, in the registration stage, this application does not, as in the prior art, only apply the multi-scale feature registration module MSFR (see [link to application details]). Figure 3 The deformation field φ is transmitted between them. i+1 Instead, it simultaneously transmits deformation features. The technical significance of this dual transmission mechanism lies in: the deformation field φ i+1 It carries the explicit expression of geometric deformation, while the deformation characteristics This implies implicit prior knowledge of registration formed during the registration process. When deformation features... The result is obtained by upsampling convolution via the input adjustment unit. and with fixed image features at the current scale. and deformation field Features of moving images after spatial transformation When concatenating along the channel dimension (see...) Figure 3 The Concat operation in the code establishes a non-linear, complementary information relationship among the three elements: Provides a structural template for the target registration state. Provides source structure information after initial alignment, while This provides the registration strategies and residual correction directions accumulated during the previous scale registration process. The interaction of these three types of heterogeneous information in the feature fusion inference unit makes the current scale registration no longer a simple refinement of the previous scale deformation, but an intelligent incremental adjustment based on an understanding of the previous scale registration strategy.

[0085] More importantly, the adaptive recursive inference strategy introduced in this application calculates the Dice coefficient using anatomical structure labels at each registration scale i as a quantitative feedback of registration quality, and decides whether to adjust the current deformation field φ based on this feedback. i The MSFR module is recursively called as prior input, thus forming a closed-loop optimization mechanism within each scale. This two-layer iterative structure, combining intra-scale recursion and inter-scale propagation, allows registration errors to be captured and corrected immediately at the scale in which they are generated, rather than passively propagating backward with the deformation field as in the traditional pyramid method. The four techniques mentioned above—bi-branch parameter decoupling, multi-layer modal-independent contrast constraints, joint propagation of deformation field and deformation features, and adaptive recursive inference—do not function independently, but rather form an organically integrated technical system: the first two ensure the modal independence of the input features as the basis for registration; the third achieves cross-scale accumulation of registration priors; and the fourth completes the adaptive optimization of the registration process through quality feedback. There is a logical progression and functional synergy among the four.

[0086] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0087] It should be noted that, through long-term in-depth research and extensive practical exploration, the inventors of this application have discovered that there are several deep-seated technical problems in existing multimodal medical image registration technologies. These problems are interconnected and jointly restrict the improvement of registration performance.

[0088] After conducting a systematic technical analysis of deep learning-based multimodal registration methods, the inventors discovered a fundamental problem in existing methods: the mixing of modality-related and modality-independent information during the feature extraction stage. In multimodal registration tasks, although there are significant differences in intensity distribution between images of different modalities, the modality-independent information they depict—such as anatomical structures, tissue boundaries, and spatial topological relationships—is inherently consistent. This modality-independent information is the true basis for establishing cross-modal correspondences. However, existing methods based on simple channel stitching or parameter-sharing encoding cannot effectively distinguish and suppress modality-specific intensity pattern information during feature extraction, resulting in the learned feature representations containing a large amount of modality-related information that is useless or even harmful to the registration task. The technical consequences of this feature mixing problem are multifaceted: on the one hand, when the model faces imaging parameter combinations or new modality pairings not seen during training, its generalization performance will significantly decrease due to over-reliance on specific intensity patterns in the training data; on the other hand, the presence of modality-related information in the feature space interferes with the accuracy of similarity measurements, making it difficult for the network to accurately assess the true correspondences between anatomical structures. The inventors deeply understand that effectively extracting modality-independent features is a key prerequisite for improving multimodal registration performance, which requires collaborative innovation at both the network architecture design and training constraint mechanisms levels.

[0089] Furthermore, through in-depth research on existing pyramid registration methods, the inventors discovered that while these methods employ a multi-scale registration strategy from coarse to fine, they suffer from severe information loss in the inter-scale information transfer mechanism. Existing methods typically only transfer the explicit expression of the geometric transformation—the deformation field—between different registration scales, while implicit prior knowledge learned by the network during the registration process, such as registration strategies, residual correction directions, and feature matching confidence, is lost with scale switching. Through theoretical analysis and experimental observation, the inventors realized that the registration process is essentially a progressively refined optimization process. In this process, the network not only needs to calculate the current deformation but also needs to accumulate and transfer empirical knowledge about how to better perform registration. When this implicit knowledge cannot be effectively transferred, subsequent scale registration loses crucial guiding information and must re-explore in a larger search space. This not only reduces registration efficiency but also makes it more prone to getting trapped in local optima. The inventors creatively proposed that, in addition to transmitting the deformation field of geometric transformation, deformation features carrying prior knowledge of registration should also be transmitted simultaneously. This dual information transmission mechanism can provide richer and more effective contextual information for registration at subsequent scales.

[0090] Furthermore, through long-term clinical application practice, the inventors noticed a systemic defect in the inherent inter-scale unidirectional transmission mechanism of the pyramid registration method: error accumulation. Within the framework of pyramid registration, the registration process starts from the smallest scale, with the registration result at each scale serving as the initial input for the next scale and propagated step-by-step. However, due to the complexity and diversity of medical images, even a well-trained network cannot guarantee optimal registration results at every scale. Once a registration deviation occurs at a certain scale, this deviation not only affects the registration quality at the current scale but also serves as erroneous prior information, propagating to all subsequent scales, leading to the amplification and accumulation of errors at each scale. In-depth analysis revealed that the root cause of this error accumulation problem lies in the lack of real-time monitoring and feedback adjustment mechanisms for registration quality in traditional methods. The registration process relies entirely on the network's single forward inference capability, failing to dynamically adjust optimization strategies based on the registration effect. While some existing technologies attempt to improve single-inference registration capability by increasing network depth or expanding network capacity, this "stacked" improvement approach not only increases computational costs but also fails to fundamentally solve the mechanism-related problem of error accumulation.

[0091] Through a systematic analysis of the aforementioned technical problems, the inventors recognized that these three issues are not isolated but rather form an interconnected chain. The contamination of modality-related information reduces the discriminative power of feature representations, thus affecting registration accuracy at each scale and creating a potential for error accumulation. The lack of prior registration knowledge weakens the ability of subsequent scales to utilize previous registration experience, resulting in a lack of coherence in the registration process. Furthermore, the existence of an error accumulation mechanism amplifies registration defects at earlier scales, ultimately leading to a decline in overall registration performance. The inventors further realized that to fundamentally solve these technical problems, a systematic technical solution is needed. This solution should achieve effective decoupling of modality information at the feature extraction level, establish a rich information accumulation mechanism at the registration transfer level, and introduce quality feedback and adaptive adjustment capabilities at the registration optimization level, thereby forming a complete technical closed loop from feature learning and registration inference to result optimization.

[0092] Based on the above in-depth research and technical insights, the inventors of this application have creatively proposed a multimodal image registration technology concept that combines dual-branch representation decoupling, joint transfer of deformation field and deformation features, and adaptive recursive reasoning. The implementation process of this application will be described in detail below through specific embodiments.

[0093] This application provides a multimodal image registration method based on representation decoupling, which is particularly suitable for registration tasks between different imaging modalities in the field of medical imaging. For example, in abdominal image analysis, CT images can provide clear information about high-density tissues such as bones, while MR images can provide rich details of soft tissues, but the spatial locations of these two modalities are often inconsistent. This application achieves accurate registration of multimodal medical images through bi-branch representation decoupling, multi-level modality-independent contrast loss constraints, and an adaptive recursive inference strategy.

[0094] In one exemplary embodiment, the registration of abdominal MR images to CT images is illustrated as an example, where the MR image serves as the moving image and the CT image as the stationary image. Those skilled in the art should understand that the technical solutions of this application are also applicable to medical image registration for other modal combinations and other anatomical locations.

[0095] refer to Figure 1 , Figure 1 The overall framework of the algorithm presented in this application is illustrated. Starting from the left, the patient's different modal images (such as the MR image at center A, the MR image at center B, the MR image at center C, and the corresponding CT image) are first processed by a dual-branch coding network for feature extraction. This dual-branch coding network contains two completely independent coding paths, processing moving and stationary images respectively, with each path outputting multiple features at different scales (labeled as F_f and F_m in the figure). Furthermore, a multi-level modality-independent contrast loss (MCLoss) is introduced during the encoding process to constrain the dual branches, ensuring that the extracted features are modality-independent. The encoded multi-scale features are then input into a registration network, which contains multiple cascaded multi-scale feature registration modules (MSFR), performing registration step by step in a coarse-to-fine order. Each MSFR module receives not only the moving and stationary image features at the current scale, but also the deformation field φ and deformation features from the previous scale, and outputs the updated deformation field and deformation features to be passed to the next scale. Finally, the deformation field (Flow) output by the registration network is applied to the original moving image to obtain a registration result aligned with the fixed image.

[0096] See Figure 4 The specific steps in this embodiment are as follows:

[0097] Step 100: Data Preprocessing

[0098] After acquiring the moving and stationary images to be registered, preprocessing is required to meet the model input requirements. This preprocessing step ensures that images from different sources and modalities have consistent intensity distribution and spatial scale, which is crucial for subsequent feature extraction and registration.

[0099] Step 110: Perform intensity normalization processing for different modal images. Specifically, for CT images, since they use Hounsfield units (HU) to represent tissue density, to highlight the display effect of abdominal organs (such as the liver), the window width and window level of the CT images are adjusted to limit the HU value range within a suitable range to display soft tissue structures. For example, the HU value range can be adjusted to the [-200, 400] range or other suitable ranges. For MR images, since different scanning equipment and scanning parameters can lead to large differences in intensity range, a quantile normalization method can be optionally used, that is, taking the higher quantile of the image pixel value as the maximum intensity value for mapping, thereby unifying the intensity range. Furthermore, in order to correct the bias field effect caused by magnetic field inhomogeneity in MR images, N4 intensity range correction or other appropriate bias field correction methods can be performed on the MR images to obtain a more uniform intensity distribution.

[0100] Step 120: Perform spatial normalization on the images. First, resample both the moving and stationary images to uniformly adjust their spatial resolution to a preset resolution. This resampling operation reduces computational costs and ensures images from different devices have a uniform physical scale. The specific resampling resolution can be set according to the actual application scenario and computing resources; for example, a moderate resolution can be chosen to balance computational efficiency and image detail preservation. Furthermore, to unify the model input size and further reduce computational load, the resampled images are center-cropped, adjusting the image size to a preset size. It should be noted that the specific cropping size can be flexibly adjusted based on the size of the region of interest and computing resources.

[0101] Step 130: Normalize the pixel values ​​of all images, mapping them to a suitable numerical range, such as the [0,1] range or other ranges that are easy for the neural network to process. Such normalization helps improve the training stability and convergence speed of the neural network.

[0102] It should be noted that during the training phase, data augmentation operations are performed on the training set images to improve the model's generalization ability and robustness. More specifically, random geometric transformations are applied to the MR images in the training set, including but not limited to random rotation, random scaling changes, and random elastic deformation, to simulate various spatial changes that may occur in clinical practice. Simultaneously, the same data augmentation strategy is applied to the CT images in the training set as to the MR images to maintain consistency in multimodal input conditions and avoid the model becoming overly dependent on specific data distributions. Optionally, specific data augmentation parameters, such as the rotation angle range and scaling change amplitude, can be adjusted according to the target application scenario.

[0103] Step 200: Two-branch multi-scale feature encoding

[0104] After preprocessing, the moving and stationary images are input into the corresponding branches of the two-branch multi-scale coding network for feature encoding. The two-branch multi-scale coding network is one of the core innovations of this application, and its design aims to extract modality-independent feature representations from images of different modalities.

[0105] Step 210: Construct a dual-branch coding network structure. This dual-branch multi-scale coding network contains two branches that are structurally identical but maintain independent network parameters. Each branch consists of multiple sub-modules connected in series, for example, three or more sub-modules connected in series, the specific number of which can be adjusted according to image resolution and computational resources. Each sub-module is implemented through a convolutional neural network with residual connections, which facilitate gradient propagation and feature reuse. Furthermore, each sub-module can employ a two-layer or multi-layer convolutional structure. For example, each sub-module performs downsampling operations simultaneously with feature extraction, such that the output feature size of the sub-module is half the input feature size or other suitable downsampling ratio, thereby forming a multi-scale feature pyramid from coarse to fine.

[0106] Step 220: Perform dual-branch independent encoding. The moving image is input into the moving image branch, and the stationary image is input into the stationary image branch. The two branches extract features from their respective input images in parallel. Since the parameters of the two branches are independent, this design allows each branch to learn the most suitable feature representation based on the characteristics of its own modality, avoiding the feature confusion problem caused by simply stitching together multimodal images in traditional methods. Each submodule of each branch outputs a feature map, ordered from smallest to largest size. The feature sequence output by the moving image branch is denoted as... The feature sequence output by the fixed image branch is denoted as The larger the superscript number, the smaller the feature size. It should be noted that the above feature labeling method is an exemplary representation, and the actual number of feature layers can be adjusted according to the network depth.

[0107] refer to Figure 2 , Figure 2This diagram illustrates the principle of the multi-modality-independent contrastive loss method used in this application. The figure shows MR and CT images of two patients (Patient_M and Patient_N), and the features extracted from these images (E_M represents moving image features, and E_F represents stationary image features). For images of different modalities of the same patient, their anatomical structures are identical, therefore their features should be brought closer together (labeled as Positive Pair). Conversely, for images of different patients, even within the same modality, their anatomical structures differ, therefore their features should be pushed apart (labeled as Negative Pair). Through this contrastive learning strategy, the model can learn modality-independent features that focus on anatomical structure rather than modality intensity.

[0108] Step 230: Calculate the multi-level modality-independent contrastive loss. To constrain the dual-branch coding network to extract modality-independent features, a contrastive loss constraint is applied to the features at different levels during the encoding process. Specifically, for each layer's output of the coding network, the corresponding contrastive loss is calculated. The overall form of the multi-level modality-independent contrastive loss is:

[0109]

[0110] Where l represents the number of layers in the coding network, L represents the total number of layers, and w L The weight coefficients represent the loss coefficients of the l-th layer. Represents moving image features based on layer l and fixed image features The calculated contrast loss.

[0111] Furthermore, for the contrast loss of each layer The information-theory-based contrastive learning loss function is adopted, and its calculation formula is as follows:

[0112]

[0113] In Formula 2, v represents the anchor feature vector, v + This represents the positive class feature vector, i.e., the image features from the same subject as the anchor point but from a different modality, (v,v) + These form positive class image pairs. This represents the negative class feature vector, i.e., image features from different subjects whose anchor points are located elsewhere. This forms a negative class image pair. N represents the number of negative class samples. τ represents a temperature parameter used to control the sensitivity of the distance between different classes. A smaller temperature parameter makes the model more sensitive to feature differences; the specific value can be adjusted based on experimental results. · represents the dot product operation, used to measure the similarity between feature vectors.

[0114] This contrastive loss design allows the model to bring image features from different modalities of the same subject closer together in the feature space during training, while pushing image features from different subjects further apart. This enables the extraction of modality-independent information (such as anatomical structures and textures) while suppressing dependence on modality-specific information (such as intensity distribution). By imposing such constraints at multiple encoding levels, the model can learn robust modality-independent representations at different levels of abstraction.

[0115] Step 300: Multi-scale feature registration

[0116] After obtaining multi-scale modality-independent features, these features are input into the registration network for step-by-step registration. The registration network adopts the pyramid registration approach, starting from the coarsest scale and gradually progressing to the finest scale. It also accumulates and refines registration information through the cascading transfer of deformation fields and deformation features.

[0117] Step 310: Construct the registration network. The registration network consists of multiple cascaded multi-scale feature registration modules (MSFR). Each MSFR module corresponds to a scale level, and registration is performed sequentially in ascending order of scale (i.e., from coarse to fine).

[0118] refer to Figure 3 , Figure 3 A schematic diagram of the multi-scale feature registration module in this application is shown. This module receives fixed image features at the current scale. Moving image features and the deformation field φ from the previous scale i+1 and deformation features As input, the module first processes the deformation features from the previous scale through convolution and upsampling operations (labeled Conv and InsNormSilu in the diagram) to obtain reference deformation features that match the current scale. Simultaneously, the deformation field of the previous scale is upsampled, and this deformation field is used to perform a spatial transformation operation (Warp) on the moving image features of the current scale to obtain initially aligned moving image features. Subsequently, fixed image features Features of the deformed moving image and reference deformation features Channel concatenation is performed, and the data is then processed by a subsequent convolutional network and a registration head network (RegHead) to finally output the deformation features at the current scale. and deformation field φ i .

[0119] Step 320: Perform the multi-scale registration process. First, at the smallest scale (coarsest scale), based on the moving image features and fixed image features at the smallest scale, the initial deformation field and deformation features are calculated using the initial MSFR module. Since this is the first scale, this module does not receive input from the previous scale, but instead performs registration calculations directly based on the features at the current scale.

[0120] In subsequent scales, each MSFR module is executed as follows:

[0121] Step 321: Input Adjustment. Receive the deformation field φ output from the previous scale. i+1 and deformation features First, the deformation features are upsampled and convolved to adjust their spatial dimensions to match the current scale features, thus obtaining the reference deformation features for the current scale. This reference deformation feature carries prior registration information from the previous scale, which helps in the registration process at the current scale. Optionally, the upsampling operation can be implemented by transposed convolution, bilinear interpolation followed by convolution, or other appropriate upsampling methods.

[0122] Step 322: Feature Deformation. The deformation field φ output from the previous scale... i+1 Perform an upsampling operation to obtain a deformation field that matches the spatial dimensions of the current scale. Using this upsampled deformation field, the moving image features at the current scale are analyzed. Perform a spatial transformation operation (Warp) to obtain pre-aligned deformed and moved image features. Specifically, spatial transformation operations are implemented through interpolation methods, such as bilinear interpolation, trilinear interpolation, or other suitable interpolation methods, to resample the moving image features according to the displacement indicated by the deformation field.

[0123] Step 323: Feature Fusion and Registration Inference. This involves fusing fixed image features at the current scale. Deformed moving image features and reference deformation features The images are concatenated along the channel dimension to form a fused feature. This fused feature simultaneously includes target information from the fixed image, information from the initially aligned moving image, and registration prior information passed from the previous scale. Subsequently, the fused feature is input into a feature processing network composed of convolutional layers for further feature extraction and fusion. Furthermore, the processed features are decoded by a registration head network (RegHead) to output the deformation features at the current scale. and deformation field φ iThe registration head network typically consists of several convolutional layers, with the number of output channels of the last layer corresponding to the dimension of the deformation field (for example, in 3D image registration, the output deformation field has three channels, corresponding to the displacements in the x, y, and z spatial directions, respectively).

[0124] Step 324: Output the deformation field φ at the current scale i and deformation features The MSFR module is passed to the next scale as input for registration at that scale. This cascading transfer mechanism allows the registration process to be progressively refined from coarse to fine, with each scale making incremental adjustments based on the previous scale, ultimately achieving accurate registration.

[0125] Step 330: After registration at all scales is completed, obtain the deformation field output at the final scale (the finest scale). Apply this deformation field to the original moving image, and resample the moving image through a spatial transformation operation to obtain a registered image that is spatially aligned with the fixed image.

[0126] Step 400: Model Training

[0127] During the training phase, a suitable loss function needs to be defined to jointly train the bi-branch multi-scale encoding network and the registration network. The overall loss function consists of several parts, which are used to constrain the learning of modality-independent features, the accuracy of registration, and the smoothness of the deformation field.

[0128] Step 410: Define the similarity loss. The similarity loss measures the degree of similarity between the registered moving image and the stationary image. For multimodal image registration, traditional loss functions based on pixel intensity differences (such as mean squared error) are ineffective because the intensity distributions of different modal images differ significantly. Therefore, this application adopts a loss function based on the local neighborhood self-similarity descriptor (MIND). This method measures similarity by comparing the local structural patterns of the images rather than absolute intensity values, making it more suitable for multimodal images. Alternatively, other similarity measurement methods suitable for multimodal registration can also be used.

[0129] Specifically, the calculation of MIND loss includes the following steps:

[0130] First, for each voxel x in the image, calculate the intensity difference D between it and its neighboring voxels in the neighborhood. p :

[0131] D p (I,x,α)=C·(I(x)-I(x+α)) 2 (Formula 3)

[0132] Where I represents the image, x represents the three-dimensional spatial position of the voxel, α represents the neighborhood offset vector, and C is a constant coefficient.

[0133] Next, the local variance V(I,x) is calculated as a normalization factor:

[0134]

[0135] in, This represents a neighborhood set, which typically includes voxels that are adjacent to the current voxel in a six-connected or twenty-six-connected sense. The specific neighborhood definition can be adjusted according to actual needs.

[0136] Then, based on the above differences and variances, a MIND vector is constructed.

[0137]

[0138] Here, Z is a normalization constant that makes the sum of the MIND values ​​in all neighborhood directions equal to 1. Each component of the MIND vector describes the similarity between the current voxel and voxels in a specific neighborhood direction, thus characterizing the local structural pattern.

[0139] Finally, regarding the deformed moving image and fixed image I fixed Calculate them at all voxel locations

[0140] The sum of the L1 distances of the MIND vectors on the vector plane is used as the similarity loss.

[0141]

[0142] Where, N vox Let ||·||1 represent the total number of voxels in the image, and ||·||1 represent the L1 norm, which is the sum of the absolute values ​​of the vector components. This loss function aims to make the registered moving image as similar as possible to the stationary image in terms of local structure.

[0143] Step 420: Define the smoothness loss of the deformation field. To avoid unreasonable folding, breaking, or other discontinuities in the deformation field, a smoothness constraint needs to be applied. The smoothness loss is achieved by penalizing the spatial gradient of the deformation field, and its calculation formula is as follows:

[0144]

[0145] Where x represents the voxel position, d represents the spatial dimension direction (x-direction, y-direction, or z-direction), and φ d (x) represents the component of the deformation field along the d direction at position x. This represents the gradient operator. This loss applies a squared penalty to the gradient of each component of the deformed field, making the deformed field as smooth and continuous as possible in space, thus ensuring the rationality and physical feasibility of the deformation.

[0146] Step 430: Construct the overall loss function. The multi-modal independent contrastive loss, similarity loss, and smoothness loss are weighted and combined to form the overall loss function for model training:

[0147]

[0148] Among them, I f Indicates a fixed image, I m Let F represent the moving image, φ represent the deformation field output by the registration network, and F represent the deformation field. f and F m λ1 and λ2 represent the fixed image features and the moving image features, respectively, and are weight coefficients used to balance the relative importance of different loss terms. It should be noted that the specific values ​​of the weight coefficients can be adjusted according to different datasets and application scenarios to obtain the optimal training results.

[0149] Step 440: Using the training set data and the defined overall loss function, jointly optimize the parameters of the two-branch multi-scale encoding network and the registration network using the backpropagation algorithm. Optionally, the Adam optimizer or other suitable gradient descent optimization algorithm can be used for parameter updates. The training process continues until the model's performance on the validation set reaches a preset standard or the loss converges. For example, appropriate training hyperparameters such as learning rate and batch size can be set to optimize the training effect.

[0150] Step 500: Adaptive Recursive Reasoning (Optional)

[0151] In the inference phase after model training, to further improve registration accuracy, this application proposes an adaptive recursive inference strategy. The core idea of ​​this strategy is to iteratively call the registration module at each registration scale and make adaptive judgments based on the registration quality index, ensuring that the optimal registration effect is achieved at that scale before passing it to the next scale, thereby avoiding the stepwise accumulation of registration errors.

[0152] Step 510: When performing the multi-scale registration process in step 300, for the current registration scale, first call the MSFR module in the conventional way to obtain the initial deformation field.

[0153] Step 520: Using the currently calculated deformation field, perform the same spatial transformation on the anatomical structure labels (such as organ segmentation labels) attached to the moving image to obtain deformed labels. Calculate the Dice coefficient or other suitable similarity metric between the deformed labels and the corresponding labels in the fixed image as a measure of the current registration quality. The Dice coefficient measures the degree of overlap between two labels; a value closer to 1 indicates a better registration effect. Optionally, other registration quality evaluation metrics can also be used.

[0154] Step 530: Compare the currently calculated Dice coefficient with the recorded reference Dice coefficient. The initial value of the reference Dice coefficient can be set to a lower value or the Dice coefficient from the previous iteration.

[0155] Step 540: If the current Dice coefficients are improved compared to the reference Dice coefficients, it indicates that the registration quality has been improved. At this point, the currently calculated deformation field is used as the prior input, and the MSFR module at the current scale is recursively called again for inference to obtain a new deformation field. Simultaneously, the reference Dice coefficients are updated to the current Dice coefficients to provide a comparison benchmark for the next iteration.

[0156] Step 550: Repeat the recursive reasoning process from steps 520 to 540 until the Dice coefficient no longer increases or a preset stopping condition is reached. For example, the stopping condition may include, but is not limited to: reaching the maximum number of iterations, the increase in the Dice coefficient being less than a preset threshold, or the registration quality reaching an acceptable level. At this point, it is considered that the optimal registration effect has been achieved at the current scale, and the recursion stops.

[0157] Step 560: Transfer the optimal deformation field and corresponding deformation features obtained at the current scale to the MSFR module at the next scale, and continue the registration at the next scale.

[0158] Through the aforementioned adaptive recursive inference strategy, the model can fully utilize available information for iterative optimization at each scale, avoiding the problem of registration error propagation and accumulation at each level in the traditional pyramid registration method, thereby achieving a more accurate registration effect overall.

[0159] This application successfully achieves modality-independent feature extraction from multimodal images by combining bi-branch independent encoding with multi-level modality-independent contrastive loss constraints. Compared with existing methods that simply stitch together multimodal images or use shared weight encoding, the bi-branch independent parameter design of this application allows each branch to be optimized for the characteristics of its respective modality, while the multi-level modality-independent contrastive loss ensures that the extracted features focus on modality-independent information such as anatomical structures, significantly improving the robustness and discriminativeness of the features.

[0160] Furthermore, this application not only transmits the deformation field in the registration network but also the deformation features simultaneously. This allows the registration at the next scale to fully utilize the prior registration information from the previous scale, achieving more effective information accumulation and refinement. Compared to existing methods that only transmit the deformation field, the joint transmission mechanism in this application provides a richer registration context, which helps improve the accuracy and stability of registration.

[0161] Furthermore, the adaptive recursive inference strategy proposed in this application, by introducing iterative optimization and quality feedback mechanisms during the inference phase, can adaptively optimize the registration effect at each scale, effectively alleviating the error accumulation problem commonly found in pyramid registration methods. This strategy is specifically optimized for specific cases, achieving excellent registration performance not only at the overall dataset level but also at the level of individual cases.

[0162] In summary, the multimodal image registration method based on representation decoupling provided in this application has strong practicality and superior technical performance, which can meet the needs of accurate registration of multimodal images in medical image analysis and provide reliable technical support for clinical diagnosis, treatment planning and other applications.

[0163] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0164] The second embodiment of this application relates to a multimodal image registration system based on representation decoupling, the structure of which is as follows: Figure 5 As shown, this multimodal image registration system based on representation decoupling includes:

[0165] The preprocessing module is used to acquire the moving image and the fixed image to be registered, and to preprocess the moving image and the fixed image.

[0166] A dual-branch multi-scale coding module is used to input the moving image and the fixed image into the corresponding branches of the dual-branch multi-scale coding network for encoding, thereby obtaining a series of moving image features and fixed image features at different scales from small to large. The dual-branch multi-scale coding network contains two branches with the same structure but independent parameters, and is constrained by a multi-layer modality-independent contrastive loss during training to extract modality-independent features.

[0167] The registration module is used to input the moving image features and fixed image features of different scales into a registration network containing multiple cascaded multi-scale feature registration modules, and perform registration step by step in order of scale from smallest to largest. In the first scale, the initial deformation field and deformation features are calculated based on the moving image features and fixed image features of the smallest scale. In subsequent scales, the input of each multi-scale feature registration module includes: the moving image features of the current scale, the fixed image features of the current scale, and the deformation field and deformation features output from the previous scale. Based on the input, the multi-scale feature registration module outputs the deformation field and deformation features of the current scale and passes them to the multi-scale feature registration module of the next scale.

[0168] The deformation application module is used to apply the deformation field output from the last scale to the moving image to obtain the registered image.

[0169] The first embodiment is a method embodiment corresponding to this embodiment. The technical details in the first embodiment can be applied to this embodiment, and the technical details in this embodiment can also be applied to the first embodiment.

[0170] The above embodiments have the following technical effects:

[0171] First, in the data preprocessing stage, window width and level adjustments and quantile normalization were applied to both CT and MR images, along with N4 intensity range correction. This effectively eliminated inconsistencies in intensity distribution caused by different modalities and acquisition devices, creating uniform input conditions for subsequent modality-independent feature extraction. Furthermore, spatial resolution was unified through resampling, and the region of interest was focused on via center cropping, reducing computational costs while preserving crucial anatomical information. During the training phase, the same data augmentation strategy was used for both MR and CT images, ensuring the model's robustness to spatial transformations and avoiding disruption of intermodal spatial correspondences due to inconsistent augmentation strategies.

[0172] In the feature encoding stage, the dual-branch multi-scale encoding network features two branches with identical structures but completely independent parameters. This allows each branch to specialize in learning specific modalities, avoiding feature confusion and optimization conflicts caused by simply stitching together images of different modalities in traditional methods. Each branch consists of multiple sub-modules connected in series with residual connections. These residual connections not only promote efficient gradient propagation but also enable hierarchical reuse of features, allowing the encoder to simultaneously capture both coarse-scale global structure and fine-scale local details. More importantly, by applying multi-level modality-independent contrastive loss constraints across multiple layers of the encoding network, the model is forced to learn to bring different modalities of the same subject closer together in the feature space, while pushing features from different subjects further apart. The profound effect of this contrastive learning mechanism lies in the fact that it not only constrains the modality independence of features at a single level of abstraction but also systematically guides the direction of feature learning at different levels of abstraction through multi-level weighted losses. This ensures that each representation level, from low-level texture to high-level semantics, focuses on modality-invariant information dimensions such as anatomical structure, while actively suppressing dependence on modality-specific information such as intensity distribution. This multi-level feature decoupling effect significantly improves the model's robustness to modal differences. Even when faced with modal combinations or imaging parameter changes that were not seen during training, the model can still maintain stable registration performance.

[0173] In the registration stage, the multi-scale feature registration module employs a pyramid architecture, achieving progressive alignment from coarse to fine through hierarchical registration from the smallest to the largest scale. At each scale, the multi-scale feature registration module receives not only the deformation field from the previous scale but also the deformation features from the previous scale. This dual information transmission mechanism produces technical effects that surpass the accumulation of simple geometric transformations. Specifically, after the input adjustment unit upsamples and convolves the deformation features from the previous scale, the resulting reference deformation features carry registration strategy information extracted during the previous scale registration process, such as implicit knowledge about which regions require greater deformation and which regions have more reliable feature matching. The feature deformation unit uses the upsampled deformation field to perform spatial transformation on the moving image features, achieving preliminary alignment at the feature level. The feature fusion inference unit concatenates the fixed image features, the deformed moving image features, and the reference deformation features, enabling the convolutional network to simultaneously perceive the target state, the current alignment state, and the registration experience from the previous scale in a unified representation space, thereby making more informed deformation field predictions. The synergistic effect of this information fusion is that the registration at the current scale is no longer a blind feature matching, but an intelligent refinement based on making full use of coarse-scale registration experience, which effectively avoids the problem of registration error accumulating and amplifying with scale transmission in traditional methods.

[0174] For similarity measurement, the MIND loss function based on local neighborhood self-similarity descriptors is adopted. Compared with traditional intensity difference measures, it can capture the local structural patterns of the image rather than absolute intensity values, making this loss function inherently robust to intensity distribution differences in multimodal images. The MIND loss constructs a structural descriptor that is insensitive to changes in imaging parameters by calculating the intensity difference between each voxel and its neighboring voxels and normalizing it with local variance. The deformation field smoothness loss ensures the continuity and physical rationality of deformation by penalizing the spatial gradient of the deformation field, avoiding unnatural folds or breaks in the deformation field. The overall loss function is a weighted combination of multimodal independent contrast loss, similarity loss, and smoothness loss, achieving a multi-objective optimization balance of feature learning, registration accuracy, and deformation smoothness.

[0175] During the inference phase, the adaptive recursive inference strategy introduces a dynamic optimization mechanism based on registration quality feedback. At each registration scale, by applying the current deformation field to the anatomical structure label of the moving image and calculating the Dice coefficient between the deformed label and the fixed image label, the model can quantitatively evaluate the registration quality at the current scale. When the Dice coefficient improves compared to the reference value, it indicates that the current registration direction is correct. At this point, the current deformation field is used as a priori input to recursively call the registration module, which can further optimize the registration effect within the same scale. The technical effect of this intra-scale recursive mechanism is that it provides multiple opportunities for iterative optimization at each scale, so that the registration process is no longer limited by the network's single forward inference capability, but gradually approaches the optimal registration state at that scale through iteration. When the Dice coefficient no longer improves, it indicates that a local optimum has been reached at the current scale. At this point, recursion stops and the optimal result is passed to the next scale. This adaptive stopping criterion allows different cases and different anatomical regions to automatically determine the number of iterations based on their registration difficulty, avoiding computational waste caused by excessive iteration and insufficient accuracy caused by underoptimization. From a system perspective, the adaptive recursive inference strategy combines the inter-scale transfer of pyramid registration with intra-scale optimization based on quality feedback, forming a two-level iterative registration framework that effectively alleviates the inherent defect of registration error accumulating step by step in traditional methods.

[0176] By combining the synergistic effects of the aforementioned technical means, the above embodiments achieve the following overall technical effects: At the feature extraction level, through the synergistic effect of dual-branch parameter decoupling and multi-layer contrast constraints, the extracted feature representations possess high modality independence and discriminability, enabling accurate capture of anatomical correspondences across intensity differences in different modalities; at the registration optimization level, through the joint transfer of deformation fields and deformation features, and the progressive refinement of the pyramid architecture, the registration process achieves effective information accumulation and error suppression from coarse to fine; at the inference adaptation level, through the quality feedback mechanism introduced by the adaptive recursive inference strategy, the registration process possesses the ability to dynamically optimize for specific cases, achieving excellent average performance not only at the overall dataset level but also at the individual case level, realizing high-precision personalized registration. The synergistic superposition of these technical effects enables the above embodiments to significantly improve the accuracy, robustness, and clinical applicability of multimodal medical image registration while maintaining good deformation field smoothness, providing more reliable technical support for applications such as medical image analysis, surgical navigation, and radiotherapy planning.

[0177] It should be noted that those skilled in the art should understand that the implementation functions of each module shown in the above-described embodiments of the multimodal image registration system based on representation decoupling can be understood with reference to the relevant descriptions of the aforementioned multimodal image registration method based on representation decoupling. The functions of each module shown in the above-described embodiments of the multimodal image registration system based on representation decoupling can be implemented by a program (executable instructions) running on a processor, or by specific logic circuits. If the above-described multimodal image registration system based on representation decoupling is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, in essence or in other words, the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any particular combination of hardware and software.

[0178] Accordingly, this application also provides a computer storage medium storing computer-executable instructions, which, when executed by a processor, implement the various method implementations of this application.

[0179] Furthermore, this application also provides a multimodal image registration system based on representation decoupling, including a memory for storing computer-executable instructions and a processor; the processor is used to implement the steps in the above-described method embodiments when executing the computer-executable instructions in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The aforementioned memory may be read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or solid-state drive, etc. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0180] It should be noted that in this patent application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this patent application, if it refers to performing an action according to an element, it means performing the action at least according to that element, including two cases: performing the action only according to that element, and performing the action according to that element and other elements. Expressions such as "multiple," "repeatedly," and "various" include two, two times, two kinds, and more than two, more than two times, and more than two kinds.

[0181] All documents mentioned in this application are considered to be incorporated in their entirety into the disclosure of this application so that they can serve as a basis for modifications if necessary. Furthermore, it should be understood that after reading the foregoing disclosure of this application, those skilled in the art can make various alterations or modifications to this application, and these equivalent forms also fall within the scope of protection claimed in this application.

Claims

1. A multi-modal image registration method based on characterizing decoupling, characterized in that, The method comprises the following steps: Step S1: obtaining a moving image and a fixed image to be registered, and preprocessing the moving image and the fixed image; Step S2: inputting the moving image and the fixed image into corresponding branches of a double-branch multi-scale encoding network for encoding to obtain a series of moving image features and fixed image features of different scales from small to large; Wherein, the double-branch multi-scale encoding network contains two branches with the same structure but independent parameters, and is constrained by multi-layer modality-independent contrastive loss during training to extract modality-independent features; Step S3: inputting the moving image features and fixed image features of different scales into a registration network containing multiple cascaded multi-scale feature registration modules, and performing registration step by step in order of scale from small to large; Wherein, in the first scale, the initial deformation field and deformation features are calculated based on the smallest scale moving image features and fixed image features; in the subsequent scales, the input of each multi-scale feature registration module includes: the moving image features of the current scale, the fixed image features of the current scale, and the deformation field and deformation features output by the last scale, and the multi-scale feature registration module outputs the deformation field and deformation features of the current scale based on the input, and passes them to the multi-scale feature registration module of the next scale; Step S4: applying the deformation field output by the last scale to the moving image to obtain the registered image.

2. The method of claim 1, wherein, The double-branch multi-scale encoding network and the registration network are pre-trained in the following way: based on the moving image and the fixed image in the training set, different scale features are obtained through the double-branch multi-scale encoding network, and multi-layer modality-independent contrastive loss is calculated, wherein the same subject's different modality image features are taken as positive pairs, and different subjects' different modality image features are taken as negative pairs; the multi-layer modality-independent contrastive loss is combined with the similarity loss and the deformation field smoothness loss into the total loss, and the parameters of the double-branch multi-scale encoding network and the registration network are updated using the total loss.

3. The method of claim 2, wherein, The multi-layer modality-independent contrastive loss is calculated by the following formula: wherein, l represents the number of layers of the encoding network, w l represents the weight of the loss of the lth layer, represents the moving image feature based on the lth layer and the fixed image feature calculated contrast loss.

4. The method of claim 1, wherein, The multi-scale feature registration module comprises: an input adjustment unit for receiving the deformation field and deformation features output by the last scale, and adjusting the deformation features to match the scale of the current scale feature by upsampling to obtain reference deformation features; a feature deformation unit for performing spatial transformation (Warp) on the moving image features of the current scale based on the deformation field output by the last scale or the deformation field adjusted in scale to obtain the deformation moving image features of the current scale; a feature fusion inference unit for combining the deformation moving image features of the current scale, the fixed image features of the current scale, and the reference deformation features, and processing them through a convolutional neural network to output the deformation field and deformation features of the current scale.

5. The method of claim 1, wherein, In the inference stage, the step S3 further comprises an adaptive recursive inference process: transforming the anatomical structure label of the moving image with the current calculated deformation field at the current registration scale, and calculating the Dice coefficient between the transformed label and the fixed image label; comparing the current Dice coefficient with the reference Dice coefficient recorded currently; when the Dice coefficient is improved compared with the reference Dice coefficient recorded currently, taking the current calculated deformation field as a prior input, recursively calling the multi-scale feature registration module to perform re-inference at the scale and updating the reference Dice coefficient; when the Dice coefficient is no longer improved or reaches a preset stopping condition, stopping recursion and passing the optimal deformation field and deformation features obtained at the current scale to the next scale.

6. The method of claim 1, wherein, the pre-processing of the images in step S1 includes: for CT images, adjusting the window width and window level to adjust the Hounsfield unit (HU) range to [-200, 400]; for MR images, taking the 99.99% quantile as the maximum intensity of the image for intensity range mapping, and performing N4 intensity range correction; resampling the moving image and the fixed image to adjust the spatial resolution of the image to 1.75x1.75x2mm; center cropping the resampled image to crop the size of the input image to 160x160x64 pixels; normalizing all image pixel values to the interval [0, 1].

7. The method of claim 1, wherein, each branch in the double-branch multi-scale encoding network is composed of three sub-modules in series; each sub-module is implemented by a double-layer convolutional neural network with residual connection; the output feature size of each sub-module is half of the input feature size of the sub-module.

8. The method of claim 4, wherein, The multi-scale feature registration module is implemented by the following specific operations: The input adjustment unit adjusts the deformation feature of the previous scale output performs an up-sampling convolution operation to output the reference deformation feature of the current scale The feature deformation unit deforms the previous scale output deformation field φ i+1 to obtain and uses the current scale moving image feature to perform a spatial transformation operation (Warp) to obtain the current scale deformed moving image feature The feature fusion inference unit fuses the fixed image features of the current scale The warped moving image features And the reference warped features Channel concatenation is performed, and the concatenated features are processed through a convolutional layer and a registration head network (RegHead) to output warped features of the current scale And the deformation field φ i .

9. The method of claim 2, wherein, Before training based on the training set images, the data enhancement operation is further included: randomly rotating, scaling and elastically deforming the MR images in the training set; the same data enhancement strategy as the MR images is used for the CT images in the training set to maintain the consistency of the input conditions.

10. A multi-modal image registration system based on characterizing decoupling, comprising: It includes: a pre-processing module for obtaining a moving image and a fixed image to be registered, and pre-processing the moving image and the fixed image; a double-branch multi-scale encoding module for inputting the moving image and the fixed image into corresponding branches of a double-branch multi-scale encoding network for encoding, to obtain a series of moving image features and fixed image features of different scales from small to large; wherein the double-branch multi-scale encoding network contains two branches with the same structure but independent parameters, and is constrained by multi-layer modality-independent contrast loss during training to extract modality-independent features; a registration module configured to input the different scale moving image features and the fixed image features into a registration network comprising a plurality of cascaded multi-scale feature registration modules, and perform registration in an order from small to large scale; wherein in the first scale, an initial deformation field and deformed features are calculated based on the smallest scale moving image features and the fixed image features; in the subsequent scales, the input of each multi-scale feature registration module comprises the moving image features of the current scale, the fixed image features of the current scale, and the deformation field and the deformed features output by the previous scale, and the multi-scale feature registration module outputs the deformation field and the deformed features of the current scale based on the input, and transmits them to the multi-scale feature registration module of the next scale; a deformation application module configured to apply the deformation field output by the last scale to the moving image to obtain the registered image.

Images