Multi-modal medical image registration method based on multi-scale prototype guided dynamic convolution

Abstract

This invention belongs to the field of medical image processing, and specifically relates to a multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution. The method includes: inputting a fixed medical image and a moving medical image into a multi-level encoder, performing upsampling at each level, and outputting a feature map of one scale at each level; in a multi-level decoder matched with a decoder, performing upsampling from level N to level I, and fusing the deformation field of level I with the moving medical image to obtain the registered image. This invention does not rely on MLP or Transformer long-distance dependency modeling, ensuring registration accuracy while also considering inference efficiency. It can provide high-precision spatial registration support for key tasks such as clinical lesion localization, multimodal data fusion, and treatment effect evaluation, and has significant clinical application value and promising prospects for widespread application.

Description

Technical Field

[0001] The technical field of this invention belongs to the field of medical image processing, and specifically relates to a multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution. Background Technology

[0002] Multimodal medical image registration is a core supporting technology in the field of medical image analysis. Its core objective is to achieve precise spatial alignment of medical images from different imaging modalities, providing spatial reference for key clinical tasks such as lesion localization, treatment efficacy evaluation, and multimodal data fusion. It has already been widely applied in clinical scenarios such as neuroimaging and abdominal imaging. In neuroimaging analysis, this technology primarily achieves alignment of T1- and T2-weighted multi-sequence magnetic resonance (MRI) images, providing a precise spatial benchmark for brain tumor contouring and quantitative analysis of cerebrovascular diseases. In abdominal imaging analysis, it can complete cross-modal registration of computed tomography (CT) and magnetic resonance (MRI), facilitating joint analysis for early disease screening and accurate staging.

[0003] Traditional medical image registration methods often employ optimization-based iterative search strategies. While these strategies achieve some registration results, they suffer from high computational costs and long execution times, often taking tens of minutes per registration cycle, making them unsuitable for practical clinical applications. In recent years, end-to-end deep learning registration frameworks based on Convolutional Neural Networks (CNNs), Visual Transformers (ViTs), and Multilayer Perceptrons (MLPs) have gradually become mainstream research, significantly improving registration efficiency. However, due to inherent differences between imaging modalities, grayscale heterogeneity caused by imaging mechanisms, and inconsistencies in cross-modal anatomical structure representations, existing deep learning registration methods still struggle to achieve high-precision and robust registration results in practical applications.

[0004] To address the core challenges of cross-modal registration, existing research methods can be broadly categorized into three types: The first type directly adopts the single-modal registration framework, introducing modality-independent similarity metrics (such as modality-independent neighborhood descriptors MIND and mutual information MI), or utilizing neural networks for similarity estimation to guide model optimization, attempting to alleviate the problem of grayscale differences across modalities; the second type relies on generative networks to achieve modality transformation, mapping the source image to the feature space of the target modality, transforming the multimodal registration problem into a single-modal registration problem, thus bypassing the direct impact of cross-modal representation differences; the third type extracts modality-independent features through learning representations or manually designing operators, establishing a structurally consistent correspondence across modalities, and eliminating the influence of modality differences at the feature level.

[0005] However, all three types of methods have inherent limitations and are difficult to solve the multimodal registration problem in complex clinical scenarios. For the first type of method, although it can partially alleviate the gray-scale differences across modalities, it cannot model the differences in anatomical structures and imaging mechanism variations between different modalities, and its performance is significantly limited in complex cross-modal scenarios such as abdominal image registration. The registration effect of the second type of method is highly dependent on the quality of the image generated by modality conversion. The artifacts introduced during modality conversion not only reduce the registration accuracy but also increase the complexity of model optimization. The third type of method is limited by the inherent defects of static convolution kernels. Static convolution kernels cannot adapt to the differences in feature distribution between different modalities, making it difficult for feature-based methods to achieve dynamic cross-modal semantic alignment and failing to fully capture the shared semantic information across modalities.

[0006] The limitations of static convolution in multimodal registration tasks are one of the core factors restricting registration performance. Although multimodal images correspond to the same anatomical structure at the semantic level, they exhibit significant heterogeneity in feature representation, making it difficult for static convolution to effectively extract corresponding local features between different modalities. Furthermore, most existing registration frameworks rely only on low-level deformation cues, failing to fully explore and utilize contextual information during the registration process. However, in multimodal image registration scenarios involving complex anatomical structures and large deformations, contextual spatial information is crucial for achieving reliable alignment. Simultaneously, existing multimodal registration methods suffer from insufficient contextual modeling when handling large deformation scenes, making it difficult to balance global structural alignment with fine-grained local feature matching. This easily leads to the accumulation of deformation field estimation errors, ultimately reducing overall registration accuracy.

[0007] In summary, existing multimodal medical image registration methods still face key challenges such as difficulty in eliminating cross-modal representation differences, the inability of static convolution to achieve dynamic feature alignment, insufficient utilization of contextual information, and insufficient registration accuracy in scenarios with large deformations. There is an urgent need to propose a novel registration framework that can synergistically alleviate modal differences, enhance cross-modal feature consistency, and fully integrate multi-scale contextual information in order to overcome the performance bottlenecks of existing technologies and meet the clinical demand for high-precision and robust multimodal image registration. Summary of the Invention

[0008] To address the problems of existing technologies, this invention proposes a multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution. The method involves inputting fixed and moving medical images into a multi-level encoder, performing upsampling at each level, and outputting a feature map of one scale at each level. In the multi-level decoder matched with the decoder, upsampling is performed step-by-step from level N to level I. The deformation field of level I is then fused with the moving medical image to obtain the registered image. The generation of the deformation field of level i specifically includes the following steps:

[0009] 101. Weight the feature maps of the mobile medical images output by the encoder at level i and level i-1 using the deformation field of the previous level. If the current level is level N, this step is not performed.

[0010] 102. Based on the feature maps of the i-th level fixed medical image and the mobile medical image, generate the i-th level fixed medical image modulation features and the mobile medical image modulation features respectively based on the multi-scale modal feature dynamic modulation module;

[0011] 103. Determine if i is 1. If yes, proceed to step 105. If not, input the feature maps of the fixed medical image and the moving medical image output at level i-1, the modulation features of the fixed medical image and the modulation features of the moving medical image at level i into the scale context flow generation module.

[0012] 104. The feature map output by the scale context flow generation module is fused with the deformation field of the previous stage by addition to obtain the deformation field of the i-th stage. Let i = i-1 and return to step 101.

[0013] 105. If i is 1, then the fixed medical image modulation features and the mobile medical image modulation features of the i-th level are convolved and then fused with the deformation field of the previous level by addition to obtain the deformation field of the first level.

[0014] Compared with existing multimodal medical image registration technologies, this invention has significant advantages and positive effects:

[0015] 1. Ben Armin significantly improves the accuracy and robustness of cross-modal registration. This invention effectively solves the semantic alignment problem caused by the heterogeneity of grayscale, texture and noise in multimodal images through self-organized prototype alignment and dynamic feature modulation. It achieves state-of-the-art performance in highly challenging multimodal registration tasks such as brain and abdomen. In particular, the alignment effect far exceeds that of traditional static convolution methods in scenarios with large deformation and complex anatomical structures.

[0016] 2. This invention integrates multi-scale context information through a scale context flow generator, which makes up for the shortcomings of single-scale modeling in traditional coarse-to-fine frameworks. It ensures the global structural consistency of the deformation field while taking into account the precise alignment of local fine-grained anatomical features, reducing the proportion of negative Jacobian determinants of the deformation field to an extremely low level and significantly enhancing physical rationality.

[0017] 3. The lightweight design of the pure CNN framework in this invention eliminates complex modules such as MLP / Transformer, improving inference efficiency by about 3 times while maintaining high accuracy. The end-to-end process does not require additional preprocessing steps such as modality conversion and pre-alignment, which greatly reduces the difficulty of engineering implementation.

[0018] 4. This invention employs a composite loss function to achieve collaborative constraints on image appearance alignment, deformation field smoothness, and cross-modal prototype consistency, adapting to various modal medical image scenarios such as CT / MRI, and providing high-precision and highly reliable registration technology support for key tasks such as clinical lesion localization and treatment response assessment.

[0019] In summary, this invention provides a multimodal medical image registration solution that balances accuracy, efficiency, and robustness, offering practical technical support for medical image analysis and clinical auxiliary diagnosis. Attached Figure Description

[0020] Figure 1 This is an overall framework diagram of a multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to the present invention.

[0021] Figure 2 This is a schematic diagram of the self-organizing prototype alignment module structure of the present invention;

[0022] Figure 3 This is a schematic diagram of the structure of the multi-scale modal feature dynamic modulation module and the scale context flow generation module of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention proposes a multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution. Fixed and moving medical images are input into a multi-level encoder, and upsampling is performed level by level, with each level outputting a feature map of one scale. In a multi-level decoder matched with the decoder, upsampling is performed level by level from the Nth level to the first level. The deformation field of the first level is then fused with the moving medical image to obtain the registered image. The generation of the i-th level deformation field specifically includes the following steps:

[0025] 101. Weight the feature maps of the mobile medical images output by the encoder at level i and level i-1 using the deformation field of the previous level. If the current level is level N, this step is not performed.

[0026] 102. Based on the feature maps of the i-th level fixed medical image and the mobile medical image, generate the i-th level fixed medical image modulation features and the mobile medical image modulation features respectively based on the multi-scale modal feature dynamic modulation module;

[0027] 103. Determine if i is 1. If yes, proceed to step 105. If not, input the feature maps of the fixed medical image and the moving medical image output at level i-1, the modulation features of the fixed medical image and the modulation features of the moving medical image at level i into the scale context flow generation module.

[0028] 104. The feature map output by the scale context flow generation module is fused with the deformation field of the previous stage by addition to obtain the deformation field of the i-th stage. Let i = i-1 and return to step 101.

[0029] 105. If i is 1, then the fixed medical image modulation features and the mobile medical image modulation features of the i-th level are convolved and then fused with the deformation field of the previous level by addition to obtain the deformation field of the first level.

[0030] like Figure 1 In one specific implementation method proposed in this embodiment, the feature encoder includes a four-level cascaded structure. Fixed medical images and moving medical images are input into the feature encoder, and the i-th level feature encoder outputs the corresponding fixed feature. and movement features Alignment is performed in the Registration Decoder, which also includes a four-level cascaded structure. When processing from the Nth level to the second level of the decoder, the i-th level uses a Multi-Scale Modality Feature Dynamic Modulation (MDM) module to obtain the corresponding modulation features based on the features output by the i-th level of the encoder. The modulation features of the i-th level and the features output by the (i-1)-th level of the encoder are used as inputs to the Scale-Context Flow Generator (SCFG) to obtain the fine registration representation of the i-th level. The deformation field of the (i+1)-th level is upsampled to the same size as the fine registration representation of the i-th level and then residual fusion is performed to obtain the deformation field of the current level. In particular, in the first level, the Scale-Context Flow Generator is replaced by a convolutional layer. This convolutional layer processes the modulation features of the first level and directly obtains the fine registration representation of the first level.

[0031] In this invention, the multi-scale modal feature dynamic modulation module relies on the cross-modal aligned semantic prototype output by the Self-organizing Prototype Alignment (SPA) module to specifically address the problem that static convolutional kernels cannot adapt to the heterogeneity of grayscale, texture, and noise in multi-modal images and are difficult to capture cross-modal shared semantic patterns. By dynamically generating adaptive convolutional kernels to modulate the multi-scale fixed / moving medical image features output by the encoder, the semantic alignment and consistency enhancement of cross-modal features are achieved, providing high-quality feature input for the subsequent accurate estimation of deformation fields.

[0032] The self-organizing prototype alignment module is a core unit for addressing the heterogeneity of cross-modal feature representations and constructing a consistent semantic benchmark across modalities. It addresses the challenges of static convolution achieving fine-grained cross-modal feature alignment and the lack of explicit alignment references in direct dynamic convolution. It integrates self-organizing clustering and symmetric contrastive learning strategies, aligning cross-modal features at the same scale from the output of the i-th layer of the encoder. and This method mines shared high-dimensional prototypes with medical anatomical semantics and achieves accurate cross-modal prototype alignment through contrastive loss. This provides clear semantic guidance for the subsequent MDM module to generate modality-adaptive convolutional kernels. Its core processing flow revolves around neighborhood information fusion, high-dimensional prototype mining, and cross-modal prototype contrastive optimization. For example... Figure 2 The data processing steps of this module include:

[0033] First, neighborhood information fusion is performed on the input moving features and fixed features respectively, and the input feature map (i.e., the moving features output by the i-th layer of the encoder) is then used. or fixed features The input feature map has dimensions H×W×D, where H, W, and D are the width, height, and depth, respectively. The neighborhood information fusion module uses a mean filter to aggregate the feature association information of the three-dimensional spatial neighborhood, obtaining the voxel features corresponding to the moving and fixed features. Each voxel feature is a... A dimensional vector, where The total number of voxels. For feature channel dimensions.

[0034] Next, the voxel features corresponding to the moving and fixed features are projected onto predefined arrays through a first shared multilayer perceptron (consisting of a 3-layer fully connected network and a ReLU activation function). 3D prototype space, The number of predefined semantic prototypes is represented as:

[0035]

[0036]

[0037] in, This represents the first shared multilayer perceptron; The original prototype for fixing medical images; To fix the voxel features of medical images; The original prototype for mobile medical imaging; Voxel features for mobile medical images.

[0038] Then, a Softmax operation is performed along the prototype dimension to generate prototype assignment weights for voxel-level moving and fixed features, respectively, realizing the probability allocation from each voxel to each semantic prototype. The weighted transformation is completed by element-wise multiplication of the weight transpose with the voxel features, yielding the weighted voxel features corresponding to the moving and fixed features, respectively, represented as:

[0039]

[0040] ;

[0041] in, Weighted voxel features for fixing medical images; For each voxel in a fixed medical image, a probability assignment matrix is ​​defined, where the elements in the i-th row and k-th column of this matrix are... Indicates the first Individuals assigned to the first The weight values ​​of each prototype; similarly, A probability assignment matrix for each voxel in a mobile medical image to its semantic prototypes, where the elements in the i-th row and k-th column of this matrix are... Indicates the first Individuals assigned to the first The weight values ​​of each prototype; This represents element-wise multiplication; This indicates finding the transpose of a matrix or vector.

[0042] Finally, a weighted average is performed on the weighted voxel features corresponding to each prototype to obtain... A cross-modal shared prototype with anatomical semantics, including:

[0043]

[0044]

[0045] in, This represents the value of the weighted voxel feature in the k-th row and i-th column of a fixed medical image; This represents the value of the weighted voxel feature in the k-th row and i-th column of a mobile medical image; This represents the k-th prototype of a fixed medical image, and the prototype vector consists of K prototypes. ; This represents the k-th prototype of a mobile medical image, and the prototype vector consists of K prototypes. ; This is a very small value used to prevent the denominator from being 0; it can be set to 10. −5 .

[0046] K prototypes from fixed medical images and moving medical images are concatenated to obtain prototype sets for fixed medical images and moving medical images, respectively. To achieve accurate alignment of semantically consistent prototypes across modalities and improve the discriminative power of different prototypes within a modality, the module introduces symmetric contrastive loss for optimized training, specifically including:

[0047] First, ℓ2 normalization is performed on both prototype sets. Prototypes with the same index across modalities are defined as positive samples, while prototypes with different indices within the same modality are defined as negative samples. Figure 2 middle , Let K represent the set of positive sample cosine similarity scores for moving and stationary medical images. Both sets have the same elements, each containing K elements. Each element is the cosine similarity between the moving medical image prototype and the stationary medical image prototype with the same index, expressed as:

[0048]

[0049] exist Figure 2 middle , This represents the set of cosine similarities between positive and negative samples of moving and stationary medical images, for the index... Positive samples are defined as positive samples, and samples from other indices of the same modality are defined as negative samples. Within a fixed medical image branch, the cosine similarity formula between positive and negative samples is:

[0050]

[0051] Similarly, in a fixed medical image branch, the cosine similarity formula between positive and negative samples is:

[0052]

[0053] in, Indicates that the index is Cross-modal cosine similarity between positive samples, i.e., the cosine similarity between mobile medical images and fixed medical images with the same prototype index in this invention; Indicates that the index is in the same modality. Positive samples and indexes are Cosine similarity of negative samples; This indicates the standard dot product operation, i.e., calculating the inner product.

[0054] For fixed medical image branches, the first The formula for the cross-entropy loss of each anchor prototype is:

[0055]

[0056] For the mobile medical imaging branch, the first The formula for the cross-entropy loss of each anchor prototype is:

[0057]

[0058] in, Indicates that the index is Fixed medical image prototype, Indicates that the index is A mobile medical imaging prototype; Indicates that the index is Fixed medical image prototype, Indicates that the index is A mobile medical imaging prototype; This is a temperature coefficient used to adjust the sharpness of the similarity distribution.

[0059] Finally, the average of the losses of all prototypes in the moving and fixed branches is taken to obtain the contrastive loss of the i-th layer encoder, as shown in the formula:

[0060]

[0061]

[0062] Where N is the number of layers in the encoder; For prototype comparison loss; This represents the contrast loss between the prototypes of a fixed medical image and a moving medical image obtained from the features output by the i-th layer of the encoder; To fix the cosine similarity between positive and negative samples of each prototype of a medical image; It represents the cosine similarity between positive and negative samples of each prototype of a mobile medical image.

[0063] Through the above design, this module can not only extract consistent semantic prototypes corresponding to human anatomical structures from cross-modal features, effectively reducing the distribution differences of cross-modal features, but also improve the semantic discriminativeness of intra-modal prototypes and avoid semantic confusion of prototypes. Its output cross-modal aligned prototype set provides a solid semantic benchmark for the dynamic convolution kernel generation of the MDM module, ensuring that the subsequent dynamically modulated cross-modal features have a high degree of semantic consistency and structural correspondence.

[0064] like Figure 3The Multi-level Modality Feature Dynamic Modulation (MDM) module is the core unit for cross-modal feature adaptive fusion and semantic consistency enhancement. It addresses the key issues that static convolutional kernels cannot dynamically adapt to the heterogeneity of cross-modal feature distributions and are prone to overfitting single-modal feature representations while ignoring the shared semantic patterns of common anatomical structures. Using the cross-modal semantic alignment prototype set output by the SPA module as explicit guidance, it achieves the adaptive generation of three-dimensional dynamic convolutional kernels through parameter mapping from prototypes to convolutional kernels. It performs layer-by-layer, targeted dynamic convolutional modulation on the same-scale cross-modal feature pairs output by the encoder, breaking the fixed representation limitations of static convolution, realizing the optimization of the structural correspondence of cross-modal features and improving semantic consistency, and finally outputting the modulated mobile features.

[0065] The core processing flow of the MDM module revolves around three key steps: prototype-guided dynamic convolution kernel generation, 3D dynamic convolution feature transformation, and feature compression and residual fusion. Each step progresses progressively, transforming the abstract cross-modal semantic prototype into concrete feature modulation operations, thus realizing the implementation of semantic prototype to feature alignment. Specifically, it includes the following steps:

[0066] First, the same-scale motion features output by the encoder are... and fixed features The input is fed into the SPA module to obtain K sets of aligned prototypes with cross-modal semantic consistency. Using these prototype sets as input, a second shared multilayer perceptron with shared weights and biases dynamically generates convolutional kernels and bias terms for each prototype to be adapted for 3D medical image registration, achieving a direct mapping from anatomical semantic prototypes to convolutional kernel parameters. The formula is as follows:

[0067]

[0068] in, The dynamically generated 3D convolution kernel can accurately capture the neighborhood feature associations of medical images in the three-dimensional space of length, width, and depth, and is adapted to the volumetric structural characteristics of medical images of the brain, abdomen, and other parts of the body. For the corresponding bias term; It is a second shared multilayer perceptron, with the same structure as the first shared multilayer perceptron; This represents the k-th prototype output of the i-th level. For the sake of simplicity, this invention will not distinguish between the processing of mobile medical images and fixed medical images. Unless otherwise specified, the corresponding features of the two images need to be processed separately.

[0069] Subsequently, using dynamically generated 3D convolution kernels and bias terms, a 3D dynamic convolution operation is performed on the features output at level i to complete the adaptive transformation and semantic extraction of cross-modal features. This prioritizes capturing shared semantic patterns (such as organ contours, lesion boundaries, and tissue textures) of homologous anatomical structures in cross-modal images such as CT / MRI and T1 / T2-MRI, effectively alleviating the problem of overfitting of static convolution to single-modal features and obtaining an intermediate feature tensor adapted to the current scale.

[0070] Next, since the intermediate feature channel dimension of the dynamic convolution output differs from that of the original input feature, it cannot directly participate in the subsequent feature fusion and deformation field estimation. Therefore, the intermediate feature is mapped and reshaped by the feature squeezing module (Squeeze(⋅)). The feature tensor dimension reshaping combined with the multilayer perceptron achieves the dimensionality reduction and fusion of the feature channels, restoring the intermediate feature to the channel dimension C of the original feature, thus completing the consistency matching of the feature dimension.

[0071] Finally, to preserve the 3D spatial details of the original features and avoid feature loss and gradient vanishing during dynamic modulation, a residual connection mechanism is introduced. The squeezed features are added element-wise to the original cross-modal features to obtain the final cross-modal features after dynamic modulation. The formula is as follows:

[0072]

[0073] in, This represents the modulation characteristics of a level i mobile or stationary medical image; The characteristics representing the moving or stationary medical image output by the encoder at level i; This represents the intermediate features of the i-th level mobile or fixed medical image.

[0074] The MDM and SPA modules form a strong collaborative working mechanism. The SPA module provides a clear cross-modal semantic benchmark for dynamic convolution kernel generation, avoiding the blind feature alignment of unguided dynamic convolution. The MDM module, on the other hand, transforms abstract semantic prototypes into concrete convolution kernel parameters, realizing the implementation of semantic prototypes into feature modulation, allowing the feature extraction capability of the convolution kernel to dynamically adapt to the cross-modal semantic prototype. Simultaneously, its multi-level modulation design is highly compatible with the decoder's coarse-to-fine registration logic: at low-resolution scales, it focuses on the dynamic fusion of global anatomical semantic features, providing a semantically consistent feature foundation for global coarse registration. At high-resolution scales, it focuses on the precise modulation of local detailed features, capturing subtle anatomical structural deformation features, achieving layer-by-layer alignment of cross-modal features from global to local. Furthermore, the module's 3D dynamic convolution design effectively captures the anatomical structural relationships in medical images in 3D space, further improving the accuracy and physical rationality of cross-modal feature modulation, enabling it to maintain excellent feature alignment performance in multi-site cross-modal registration tasks such as brain T1 / T2-MRI and abdominal CT / MRI.

[0075] The training process of this invention is shown in Table 1.

[0076] Table 1

[0077] Multi-scale prototype-guided dynamic convolution multimodal medical image registration algorithm flow Input: Fixed medical image Mobile medical imaging Number of prototypes , scale number Loss weight Output: Registered moving image Final deformation field 1. Initialize encoder / decoder network parameters, SPA / MDM / SCFG module parameters, and set the Adam optimizer (learning rate). ); 2: Yes Perform normalization (map pixel values ​​to) ), resampling (uniform resolution) preprocessing; 3: Multi-scale feature encoding: Encoding the preprocessed features... Input 4-level encoder 4: Self-organizing prototype alignment (SPA): 4.1: For the feature map output from each stage of the encoder, voxel features are obtained through neighborhood fusion, and then projected onto a shared MLP. Dimensional prototype space, generate prototypes and assign weights; 4.2: Calculation A semantic prototype for cross-modal alignment; 4.3: Calculate the prototype contrast loss; 5: Multi-scale Modal Feature Dynamic Modulation (MDM): 5.1: For each scale, the prototype vectors are dynamically used to generate 3D convolution kernels and biases via a shared MLP; 5.2: Perform dynamic convolution to obtain intermediate features; 5.3: The feature squeezing module maps back to the original channel dimension, and the residual connection is used to output the modulation feature; 6: Scale Context Stream Generation (SCFG) (coarse-to-fine iteration): 6.1: Initialize the coarsest scale ( Deformation field 6.2: From arrive Reverse iteration: 6.2.1: If The deformation field of the previous stage is upsampled to the current scale resolution. The i-th level modulation feature is distorted to obtain the global coarse alignment feature. The two feature map channels are concatenated and then fused by convolution. 6.2.2: The current scale deformation field is inferred by the registration head based on the fused feature. 7: Registration result generation: using the final deformation field right Perform spatial transformation to obtain the registered image. 8: Calculation of Composite Loss: 8.1: Similarity Loss 8.2: Smoothness Loss 8.3: Total Loss 9: Backpropagate to update all module parameters, repeat steps 2-8 until the training epochs reach 500; 10: Output the registered image. and final deformation field .

[0078] The similarity loss between the registered image and the fixed medical image is obtained. Represented as:

[0079]

[0080] in, Represents the registered image In the middle, the neighborhood window is set with pixel p as the center. Within the neighborhood modality independent neighborhood descriptor, neighborhood window It can be set to sizes such as 3×3×3 and 5×5×5; Represents the registered image A set of pixels; Representing fixed medical images A modal-independent neighborhood descriptor centered at pixel p within a defined neighborhood; Represents the registered image The center pixel in the registered image As a neighborhood modality-independent descriptor, it uses the image center pixel as the center and the entire image as the neighborhood window. Calculate the modally independent neighborhood descriptor; Representing fixed medical images The center pixel in a fixed medical image As a neighborhood modality-independent neighborhood descriptor within the neighborhood range; This indicates the calculation of the L2 norm.

[0081] If the image Centered on pixel p, For a neighborhood window, its modality-independent neighborhood descriptor is represented as:

[0082]

[0083] in, For image The pixel value of pixel p; Representing an image offset from pixel p The pixel value at that pixel; The average value of pixels within the neighborhood window; The standard deviation of pixel values ​​within the neighborhood window; Represents the neighborhood window The set consisting of all pixels except the center pixel p. The number of pixels in the neighborhood is the neighborhood set of pixel p in the neighborhood window.

[0084] Smoothness loss of deformation field Represented as:

[0085]

[0086] Where N is the number of layers in the decoder; This indicates the gradient of the deformation field output by the i-th layer of the decoder; To calculate the L2 norm.

[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution, characterized in that, Fixed medical images and mobile medical images are input into a multi-level encoder, and upsampling is performed step by step. Each level outputs a feature map of one scale. In a multi-level decoder matched with the decoder, upsampling is performed step by step from the Nth level to the first level. The first-level deformation field is then fused with the mobile medical image to obtain the registered image. The generation of the i-th level deformation field specifically includes the following steps:

101. Weight the feature maps of the mobile medical images output by the encoder at level i and level i-1 using the deformation field of the previous level. If the current level is level N, this step is not performed.

102. Based on the feature maps of the i-th level fixed medical image and the mobile medical image, generate the i-th level fixed medical image modulation features and the mobile medical image modulation features respectively based on the multi-scale modal feature dynamic modulation module; 103. Determine if i is 1. If yes, proceed to step 105. If not, input the feature maps of the fixed medical image and the moving medical image output at level i-1, the modulation features of the fixed medical image and the modulation features of the moving medical image at level i into the scale context flow generation module.

104. The feature map output by the scale context flow generation module is fused with the deformation field of the previous stage by addition to obtain the deformation field of the i-th stage. Let i = i-1 and return to step 101.

105. If i is 1, then the fixed medical image modulation features and the mobile medical image modulation features of the i-th level are convolved and then fused with the deformation field of the previous level by addition to obtain the deformation field of the first level.

2. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 1, characterized in that, The process of generating modulation features by the multi-scale modal feature dynamic modulation module includes: For the current scale Feature map of fixed medical images Feature maps of mobile medical images The corresponding voxel features are obtained through neighborhood fusion, and the obtained voxel features are projected onto the first shared multilayer perceptron. Dimensional prototype space, generate corresponding prototypes and assign weights; The voxel features of fixed and mobile medical images are weighted using the obtained prototype assignment weights to obtain the corresponding semantic prototypes. The semantic prototypes of fixed medical images and moving medical images are respectively input into the second shared multilayer perceptron to generate corresponding 3D convolution kernels and biases; Based on the generated 3D convolution kernel and bias, the current scale is... Feature map of fixed medical images Feature maps of mobile medical images Perform dynamic convolution to obtain the corresponding intermediate features; The intermediate features are mapped back to the original channel dimension using the feature compression module, and then compared with the feature maps of the corresponding fixed medical images. Feature maps of mobile medical images Perform parameter concatenation to output the modulation features of fixed medical images and the modulation features of moving medical images.

3. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 1, characterized in that, The data processing by the i-th level scale context stream generation module includes: The feature maps of the fixed medical image and the moving medical image output at level i-1 are concatenated and then convolved to obtain the representation at level i-1. The fixed medical image modulation features and the moving medical image modulation features of level i are concatenated and then convolved to obtain the point-pair representation of level i. The (i-1)th level representation and the point-to-point representation of the i-th level are concatenated and fused by convolution to form the final feature of the i-th level. The final feature of level i is input into a convolutional layer for processing to obtain the deformation field of level i.

4. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 1, characterized in that, The loss function used when training the encoder and decoder is expressed as: in, This is the total loss function during training; To obtain the similarity loss between the registered image and the fixed medical image, for Weighting coefficients; For the smoothness loss of the deformation field, for Weighting coefficients; For prototype comparison loss, for The weighting coefficients.

5. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 4, characterized in that, Prototype contrast loss Represented as: Where N is the number of layers in the encoder; This represents the contrast loss between the prototypes of a stationary medical image and a moving medical image obtained from the features output by the i-th layer of the encoder; To fix the cosine similarity between positive and negative samples of each prototype of a medical image; This represents the cosine similarity between positive and negative samples of each prototype in a mobile medical image; K is the number of prototypes.

6. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 4, characterized in that, The cosine similarity between positive and negative samples of each prototype in both stationary and mobile medical images is expressed as follows: in, Indicates the calculation of the inner product; Indicates that the index is Fixed medical image prototype, Indicates that the index is A mobile medical imaging prototype; Indicates that the index is Fixed medical image prototype, Indicates that the index is A mobile medical imaging prototype; This is the temperature coefficient.

7. A multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claims 1, 2, and 6, characterized in that, The medical image prototype is represented as: in, This represents the value of the k-th row and i-th column of the weighted voxel features in a fixed medical image. This represents the weight value assigned to the k-th prototype of the i-th voxel in a fixed medical image; This represents the value of the weighted voxel feature in the k-th row and i-th column of a mobile medical image. This represents the weight value assigned to the k-th prototype of the i-th voxel in a mobile medical image; This represents the k-th prototype of a fixed medical image; This represents the k-th prototype of a mobile medical image; It is a local minimum.

8. The multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 4, characterized in that, The similarity loss between the registered image and the fixed medical image is obtained. Represented as: in, Represents the registered image The modal independent neighborhood descriptor is defined within a specified neighborhood centered at pixel p. Represents the registered image A set of pixels; Representing fixed medical images A modal-independent neighborhood descriptor centered at pixel p within a defined neighborhood; Represents the registered image The center pixel in the registered image As a neighborhood modality-independent neighborhood descriptor within the neighborhood range; Representing fixed medical images The center pixel in a fixed medical image As a neighborhood modality-independent neighborhood descriptor within the neighborhood range; This indicates the calculation of the L2 norm.

9. A multimodal medical image registration method based on multi-scale prototype-guided dynamic convolution according to claim 4, characterized in that, Smoothness loss of deformation field Represented as: Where N is the number of layers in the decoder; This indicates the gradient of the deformation field output by the i-th layer of the decoder; To calculate the L2 norm.

Images