A multi-modal medical image registration method based on feature decoupling and reuse
By employing a hierarchical feature decoupling and reuse method, structural and appearance features of multimodal medical images are extracted and fused to predict deformation fields, thus solving the problem of information waste in existing technologies and achieving high-precision and robust multimodal image registration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-24
- Publication Date
- 2026-07-14
AI Technical Summary
Existing multimodal medical image registration methods based on feature decoupling struggle to achieve complete decoupling in high-dimensional feature spaces, resulting in the loss of a large amount of geometric structural information in modality-related appearance features, which limits registration accuracy and robustness.
A weight-sharing encoder network is used to extract multi-level features. Combined with a hierarchical feature decoupler and a modal feature reuse module, structural correlation features and modal correlation features are extracted through a block-based cross-attention mechanism to predict the main deformation field and auxiliary deformation field. The deformation field is refined layer by layer to achieve secondary mining and utilization of information.
It significantly improves the accuracy and robustness of multimodal medical image registration, avoids information waste, and enhances anatomical alignment performance under complex nonlinear deformation.
Smart Images
Figure CN122391311A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical image processing technology, specifically relating to a multimodal medical image registration method based on feature decoupling and reuse. Background Technology
[0002] Multimodal medical image registration aims to spatially align medical images from different imaging modalities (such as CT, MRI, PET, etc.) to fuse complementary anatomical and functional information, providing crucial support for clinical diagnosis, surgical navigation, and efficacy evaluation. Because different modalities exhibit significant differences in grayscale distribution and texture features for the same tissue structure, traditional registration methods relying on image grayscale similarity face considerable challenges. Therefore, developing high-precision and robust multimodal registration methods has significant clinical value.
[0003] In existing technologies, deep learning-based multimodal medical image registration methods are mainly divided into indirect registration and direct registration. Indirect registration methods "translate" the source modality image into the target modality through generative networks, reducing the multimodal problem to a single modality problem. However, its registration accuracy is highly dependent on the quality of the generated image; artifacts or structural distortions can directly transmit and amplify the registration error. Direct registration methods extract modality-insensitive geometric features by designing modality-insensitive similarity metrics or improving the network structure. Among these, feature decoupling methods have received widespread attention: these methods decompose image features into modality-invariant structural features and modality-specific appearance features, and use structural features to predict deformation fields, thereby avoiding interference from modality differences.
[0004] However, existing registration methods based on feature decoupling generally suffer from the following problems: achieving complete decoupling in high-dimensional feature spaces is extremely difficult, and the parts classified as modality-related appearance features still retain a large amount of geometric structural information beneficial to the registration task. Current methods typically discard these appearance features directly, which not only wastes information but also limits the upper limit of deformation field prediction accuracy. Therefore, how to fully mine and reuse the structural information implicit in modality-related features to further improve the accuracy of multimodal image registration is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] To address the problems existing in the background art, one aspect of the present invention provides a multimodal medical image registration method based on feature decoupling and reuse, comprising:
[0006] S1: Obtain a multimodal medical image dataset, wherein the multimodal medical image dataset includes image pairs consisting of a first modal medical image and a second modal medical image, wherein the first modal medical image is a fixed image and the second modal medical image is a moving image;
[0007] S2: Utilize a weight-sharing encoder network to extract multi-level hierarchical features from both moving and stationary images;
[0008] S3: For both moving and stationary images, construct a hierarchical feature decoupler, a modal feature reuse module, and a deformation field prediction module for the hierarchical features at each level.
[0009] S4: After applying the deformation field of the previous level to the hierarchical features of the moving image at the current level, it is input into the hierarchical feature decoupler corresponding to the current level, along with the hierarchical features of the fixed image at the current level, to separate the structural features of the fixed image, the structural features of the moving image, the appearance features of the fixed image, and the appearance features of the moving image at the current level.
[0010] S5: Input the fixed image structure features, moving image structure features, fixed image appearance features, and moving image appearance features of the current level into the modal feature reuse module corresponding to the current level, and extract the structural association features and modal association features through the block cross attention mechanism;
[0011] S6: Input the structural correlation features and modal correlation features into the deformation field prediction module corresponding to the current level, respectively, to predict the main deformation field and auxiliary deformation field of the current level; add the features of the main deformation field and auxiliary deformation field to obtain the initial deformation field of the current level;
[0012] S7: Apply the deformation field of the previous level to the initial deformation field of the current level to obtain the deformation field of the current level;
[0013] S8: Use the deformation field of the last layer as the final deformation field, and perform spatial transformation on the moving image through the final deformation field to obtain the registered moving image.
[0014] Another aspect of the present invention provides a multimodal medical image registration system based on feature decoupling and reuse, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the multimodal medical image registration method based on feature decoupling and reuse.
[0015] Another aspect of the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the aforementioned multimodal medical image registration method based on feature decoupling and reuse.
[0016] The present invention has at least the following beneficial effects
[0017] This invention separates structural and apparent features using a hierarchical feature decoupling device. It then leverages a block-based cross-attention mechanism in the modal feature reuse module to effectively extract geometrically relevant information from the apparent features. This information is then fused with the structural features to jointly predict the primary and secondary deformation fields, thus enabling secondary mining and utilization of potential structural information from "discarded" features. This method not only avoids wasting useful information but also significantly improves registration accuracy and robustness under complex nonlinear deformations through layer-by-layer refinement of the multi-level deformation fields, exhibiting higher anatomical alignment performance compared to traditional decoupling registration methods. Attached Figure Description
[0018] Figure 1 This is a schematic diagram of the structural model of the present invention;
[0019] Figure 2 This is a schematic diagram of the feature extraction branch structure of the encoder network in this invention;
[0020] Figure 3 This is a schematic diagram of the model structure of the hierarchical feature decoupling device in this invention;
[0021] Figure 4 This is a schematic diagram of the model structure of the reuse module of the present invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] Please see Figure 1 One aspect of the present invention provides a multimodal medical image registration method based on feature decoupling and reuse, comprising:
[0024] S1: Obtain a multimodal medical image dataset, wherein the multimodal medical image dataset includes image pairs consisting of a first modal medical image and a second modal medical image, wherein the first modal medical image is a fixed image and the second modal medical image is a moving image;
[0025] In this embodiment, multimodal MRI image registration of the brain is used as an example. T1 and T2 modal MRI images from the publicly available BraTS2018 dataset are obtained as a multimodal medical image dataset. The T1 modal images are used as the first modal fixed images, and the T2 modal images are used as the second modal moving images. The original image size is 155×240×240 voxels, with a spatial resolution of 1 mm and isotropic. To simulate spatial misalignment in clinical settings, elastic deformation and Gaussian smoothing are randomly applied to the T2 modal moving images, generating misaligned image pairs. All images are then centrally cropped to a uniform size of 160×192×128 voxels. The processed image pairs are divided into training, validation, and test sets in a 7:1.5:1.5 ratio. The training set is used for network parameter optimization, the validation set for hyperparameter adjustment, and the test set for final registration performance evaluation. Through these steps, a multimodal medical image dataset is constructed for subsequent feature extraction and registration learning.
[0026] S2: Utilize a weight-sharing encoder network to extract multi-level hierarchical features from both moving and stationary images;
[0027] Please see Figure 2 Preferably, the weight-sharing encoder network includes two feature extraction branches with shared weights, each feature extraction branch including four cascaded convolutional blocks; each convolutional block consists of cascaded convolutional layers and a ReLU activation function; the output features of all convolutional blocks in the two feature extraction branches are extracted and arranged in ascending order of scale to obtain a hierarchical feature set of the moving image. and the hierarchical feature set of a fixed image ;feature and Represents moving and stationary images. Hierarchical features at each level, .
[0028] In this embodiment, a weight-sharing encoder network is constructed. This network contains two feature extraction branches with identical structures and shared parameters, used to process stationary and moving images respectively. Each feature extraction branch consists of four cascaded convolutional blocks; the specific structure of each convolutional block is as follows:
[0029] The first convolutional block contains a 3×3×3 convolutional layer (stride of 1) and a ReLU activation function, with 16 output channels. Since the stride is 1, this layer does not change the feature map spatial size, and the output size remains 160×192×128.
[0030] The second convolutional block contains two 3×3×3 convolutional layers, each followed by a ReLU activation function. The first convolutional layer has a stride of 2 (for downsampling), and the second convolutional layer has a stride of 1. The output has 32 channels and a feature map size of 80×96×64.
[0031] The third convolutional block has the same structure as the second convolutional block, containing two 3×3×3 convolutional layers (first stride 2, second stride 1) and ReLU, with 64 output channels and an output size of 40×48×32.
[0032] The fourth convolutional block has the same structure as the second convolutional block, containing two 3×3×3 convolutional layers (first stride 2, second stride 1) and ReLU, with 128 output channels and an output size of 20×24×16.
[0033] Features from the outputs of the four convolutional blocks in the fixed image branch are extracted and arranged in ascending scale (i.e., from high resolution to low resolution) to obtain the hierarchical feature set {F1, F2, F3, F4} for the fixed image. F1 is the output of the first convolutional block (160×192×128×16, highest resolution), F2 is the output of the second convolutional block (80×96×64×32), F3 is the output of the third convolutional block (40×48×32×64), and F4 is the output of the fourth convolutional block (20×24×16×128, lowest resolution). Similarly, the hierarchical feature set {M1, M2, M3, M4} for the moving image is extracted from the moving image branch. Since the two branches share weights, it ensures that the fixed and moving images have a consistent feature representation space at the same level, providing a multi-scale feature foundation for subsequent layer-by-layer registration.
[0034] S3: For both moving and stationary images, construct a hierarchical feature decoupler, a modal feature reuse module, and a deformation field prediction module for the hierarchical features at each level.
[0035] Please see Figure 3 S4: After applying the deformation field of the previous level to the hierarchical features of the moving image at the current level, it is input into the hierarchical feature decoupler corresponding to the current level, separating the structural features of the fixed image, the structural features of the moving image, the appearance features of the fixed image, and the appearance features of the moving image at the current level.
[0036] Preferably, step S4 includes:
[0037] S41: The deformation field of the previous level The displacement amplitude is magnified by 2 times, and then the deformation field is obtained by trilinear interpolation upsampling. In the first layer, there is no upper-layer deformation field, so it is directly taken as... Zero field;
[0038] S42: Based on the deformation field Using spatial transformation networks Hierarchical features of the current layer of the moving image Features are obtained by deformation Then, the deformed hierarchical features Hierarchical features of the current layer fixed image The input hierarchical feature decoupler separates the fixed image structure features of the current level. Moving image structural features Fixed image appearance features and moving image appearance features .
[0039] Preferably, the hierarchical feature decoupler includes: a first AMC module, a second AMC module, an AGS module, and four convolutional blocks; each convolutional block includes a cascaded convolutional layer, an IN normalization layer, and a ReLU activation function; the feature processing of the hierarchical feature decoupler includes:
[0040] S401: Let the two input features of the hierarchical feature decoupler be... and , , , ;
[0041] S402: Features and characteristics The first and second AMC modules are respectively input for context feature extraction to obtain features. and characteristics ;
[0042] S403: Features and characteristics The AGS module is used to extract the structural features of modal sharing, resulting in the feature... and characteristics ;
[0043] S404: Features and characteristics The image structure features are obtained by processing them through two convolutional blocks respectively. and moving image structural features ;
[0044] S405: Features and characteristics After feature subtraction, the image is further processed through a convolutional block to obtain fixed image appearance features. ;
[0045] S406: Features and characteristics After feature subtraction, the image is further processed through a convolutional block to obtain the apparent features of the moving image. .
[0046] Preferably, the first AMC module and the second AMC module have the same network structure, both including convolutional layers and multiple layers with different porosities. Convolution; the context feature extraction process in the first and second AMC modules includes: using multiple modules with different dilatation rates Convolutional layers extract multi-scale contextual information from the input features, resulting in contextual features at multiple scales. These contextual features are then concatenated and processed through a convolutional layer to obtain the final feature. and characteristics .
[0047] In this embodiment, three-dimensional dilated convolutions with dilation rates of 1, 2, and 4 are used to extract multi-scale contextual features in parallel. All scale contextual features are concatenated and fused through a 1×1×1 convolutional layer to output: features. and characteristics .
[0048] Preferably, the AGS module includes: three convolutional processing layers, each convolutional processing layer including a cascaded convolutional layer and an IN normalization layer; the feature processing procedure of the AGS module includes:
[0049] For input features and characteristics Features are obtained by processing through two convolutional layers respectively. and characteristics ;
[0050] Features and characteristics After concatenation, the data is fed into a convolutional processing layer to obtain the features. ;
[0051] Features The feature is obtained by uniformly splitting the channel along its dimension into two parts. and characteristics :
[0052]
[0053] in, Indicates features It is evenly divided into two features along the channel dimension;
[0054] Based on characteristics and characteristics The relative weight matrix of features is calculated using Softmax normalization. According to characteristics and characteristics The relative weight matrix of features is calculated using Softmax normalization. : , ; ; ; ; ;in, Represents an exponential function;
[0055] Relative weight matrix of features and characteristics The output features are obtained by performing element-wise multiplication. ; Relative weight matrix of features and characteristics The output features are obtained by performing element-wise multiplication. .
[0056] In this embodiment, the current-level moving image features are pre-distorted and aligned by the upper-level deformation field. Then, combined with the multi-scale context extraction and adaptive weight structure separation mechanism of the hierarchical feature decoupler, the modality-independent structural features and modality-related appearance features can be accurately decoupled. This avoids the waste of effective structural information in appearance features and improves the purity and reliability of feature decoupling. At the same time, the layer-by-layer refinement method can be adapted to complex nonlinear deformation scenarios, laying a stable feature foundation for subsequent high-precision deformation field prediction.
[0057] S5: Input the fixed image structure features, moving image structure features, fixed image appearance features, and moving image appearance features of the current level into the modal feature reuse module corresponding to the current level, and extract the structural association features and modal association features through the block cross attention mechanism;
[0058] Preferably, step S5 includes:
[0059] S51: Fix image structural features and moving image structural features By adding the features together, we obtain the fused structural features. ;
[0060] S52: Integrating structural features Fixed image appearance features and moving image appearance features Each feature is divided into multiple non-overlapping feature blocks using the same block partitioning method;
[0061] S53: Map the feature blocks with fused structural features to Q vectors through a linear layer, and map the image blocks with fixed image appearance features to first K vectors and first V vectors respectively through a linear layer; perform block-based cross-attention calculation based on Q vectors, first K vectors, and first V vectors to obtain first attention-weighted features; add the attention-weighted features to the feature blocks with fixed image appearance features to obtain enhanced first feature blocks; merge all enhanced first feature blocks into blocks to obtain modality association features;
[0062] S54: Map the feature blocks of the appearance features of the moving image to the second K vector and the second V vector respectively through a linear layer; perform block cross-attention calculation based on the Q vector, the second K vector and the second V vector to obtain the second attention weighted feature; add the second attention weighted feature to the feature blocks of the appearance features of the moving image to obtain the enhanced second feature block; merge all the enhanced second feature blocks into blocks to obtain the structural association feature.
[0063] In this embodiment, specifically in step S5, the fixed image structure features obtained at the current level are first... With moving image structural features Element-by-element addition yields the fused structural features. Subsequently, Fixed image appearance features and the appearance features of moving images Using the same block partitioning method (e.g., dividing into non-overlapping 3D blocks of 5×6×5 voxels), a one-to-one block-level correspondence is ensured among the three: the fused structural feature block, the fixed image appearance feature block, and the moving image appearance feature block at the same spatial location correspond to the same local region of the image. In the block-based cross-attention calculation, the fused structural feature block is used as the query source and mapped to a query vector Q through a linear layer. Simultaneously, a fixed image appearance feature block with the same spatial location as the structural feature block is taken and mapped to a key vector (first K vector) and a value vector (first V vector) through a linear layer. Cross-attention calculation is performed to obtain the first attention-weighted feature, which is then added to the original fixed image appearance feature block and merged into a modality association feature. Similarly, the Q vector generated by the structural feature block at the same location is combined with the second K vector and the second V vector obtained by mapping the moving image appearance feature block at the same location to calculate the second attention-weighted feature, which is then superimposed on the moving image appearance feature block and merged to obtain the structural association feature. This block-based cross-attention mechanism, based on spatial correspondence, can accurately extract geometrically relevant information from corresponding local regions of apparent features, providing more spatially targeted structural clues for subsequent deformation field prediction.
[0064] S6: Input the structural correlation features and modal correlation features into the deformation field prediction module corresponding to the current level, respectively, to predict the main deformation field and auxiliary deformation field of the current level; add the features of the main deformation field and auxiliary deformation field to obtain the initial deformation field of the current level;
[0065] In this embodiment, the structural correlation features and modal correlation features output in step S5 are respectively input into the deformation field prediction module corresponding to the current level. This module consists of a 3×3×3 convolutional layer with 3 output channels (corresponding to the three directions of the three-dimensional spatial displacement field), predicting the main deformation field and auxiliary deformation field of the current level respectively. The main deformation field mainly reflects the global geometric transformation driven by the fused structural features, while the auxiliary deformation field captures the local structural details mined from the modal correlation features. Subsequently, the two are element-wise summed on a voxel-by-voxel basis to obtain the initial deformation field of the current level. This fusion strategy retains the registration dominance of structural features while making full use of the reusable structural information in the appearance features, thereby achieving complementary enhancement of main and auxiliary information at the initial deformation field level.
[0066] S7: Apply the deformation field of the previous level to the initial deformation field of the current level to obtain the deformation field of the current level;
[0067] Preferably, step S7 includes: transferring the deformation field from the previous level... The displacement amplitude is magnified by 2 times, and then the deformation field is obtained by trilinear interpolation upsampling. According to the deformation field Using spatial transformation networks Initial deformation field at the current level Perform deformation to obtain the initial coarse deformation field of the current level; then combine the initial coarse deformation field of the current level with the initial deformation field of the current level. By summing the features, the deformation field of the current level is obtained. .
[0068] In a specific embodiment of step S7, the previous level (the first) is first obtained. The deformation field calculated for the layer) The displacement amplitude is magnified by a factor of 2 to match the spatial scale of the current layer feature map, and then upsampled to the resolution of the current layer using trilinear interpolation to obtain the deformation field. Next, the Spatial Transformation Network (STN) is used according to... The initial deformation field of the current level obtained in step S6 Perform spatial deformation: specifically, with As a sampling grid pair Differentiable resampling is performed to obtain the "initial coarse deformation field". This operation achieves the transfer and initialization from the coarse deformation field of the previous level to the fine mesh of the current level. Finally, the initial coarse deformation field is compared with the initial deformation field of the current level. To perform element-wise addition, that is... Thus, the first The final deformation field of the layer For the first level (i=1), since there is no deformation field in the previous level, it is directly... Set to zero field, so that By employing the aforementioned layer-by-layer transmission and stacking strategy, the continuity and precision of the deformation field from coarse resolution to fine resolution are ensured, effectively improving the smoothness and accuracy of the final registration deformation.
[0069] S8: Use the deformation field of the last layer as the final deformation field, and perform spatial transformation on the moving image through the final deformation field to obtain the registered moving image.
[0070] In this embodiment, the deformation field of the last layer is taken as the final deformation field. This serves as the final deformation field. The deformation field has the same spatial dimensions as the original moving image (e.g., 160×192×128 voxels), with each voxel location recording a three-dimensional displacement vector from the moving image to the stationary image. Subsequently, a Spatial Transformation Network (STN) is used to... Differentiable spatial transformation is performed on the original moving image to obtain the registered moving image. The registered moving image achieves pixel-level spatial alignment with the fixed image FF in terms of anatomical structure, and can be directly used for subsequent clinical applications such as multimodal fusion analysis, lesion localization, or surgical navigation. STN stands for Spatial Transformer Network, which is used to apply differentiable spatial deformation to feature maps or images.
[0071] Preferably, the loss function used in training the AI model in steps S1 to S8 includes:
[0072]
[0073]
[0074]
[0075]
[0076]
[0077]
[0078] in, Represents the total loss function; , and Indicates the weighting parameter; This represents the comparison decoupling loss function; Indicates the regularization loss; This indicates the MIND similarity loss; To represent feature similarity calculation, cosine similarity or negative Euclidean distance is usually used. Indicates the temperature coefficient; and This represents two distinct features among the current level's fixed image structural features, moving image structural features, fixed image appearance features, and moving image appearance features; and The positive sample pairs are considered positive sample pairs, and the remaining combinations are considered negative sample pairs. Indicates the final deformation field; Represents the deformation field In voxels The displacement vector of the position; Represents the spatial gradient operator; Represents the deformation field In voxels Spatial gradient of location; express The square of the L2 norm; Indicates the position of a voxel in the image; This represents the offset vector, indicating the distance within the neighborhood relative to the center. The displacement; Represents a set of neighborhood offsets (e.g., several preset offset directions in a 3D image, such as...). ; Indicates the fixed image at position Offset at location MIND features; This indicates that the registered moving images are at the same spatial location. Offset at location MIND features; Represents the square of the Euclidean distance; Indicates the image at position The intensity value; Indicates the image at position The intensity value; This represents a local variance estimate; Represents very small positive numbers (e.g.) ), to prevent the denominator from being zero; This represents the total number of offset vectors in the neighborhood offset set.
[0079] In this embodiment, the weighting parameter is set as follows: , , ; The values in this set are determined through a grid search on the validation set, achieving a balance between registration accuracy and deformation smoothness. A neighborhood offset set is defined. For 3D images, a 26-neighborhood direction centered on the central voxel is selected (i.e., the offset vector). Meanwhile, to reduce computational load, a 6-neighborhood can also be used. (Direction). This embodiment uses a 6-neighborhood, that is... .
[0080] Comparing the decoupling loss function calculation formula, for the i-th layer, the feature set contains four features. Positive sample pairs are Negative sample pairs are , , , and .
[0081] Using the Adam optimizer, the initial learning rate is set to... The batch size is 1 (full 3D image), and the training is performed for 200 epochs. The average loss is calculated on the validation set every 10 epochs, and the best model is saved. By minimizing the total loss, the network can simultaneously achieve high-precision registration, smooth deformation fields, and clean feature decoupling.
[0082] The loss function, through the synergistic optimization of MIND similarity loss, regularization loss, and contrastive decoupling loss, significantly improves the accuracy and robustness of multimodal medical image registration: MIND loss drives registration with modality-independent local structural descriptors, overcoming the limitations of grayscale differences between different imaging modalities; contrastive decoupling loss guides the network to achieve clean feature separation by bringing cross-modal structural features closer together and pushing structural features and appearance features further apart, while allowing residual structural information in appearance features to be further mined by the modal feature reuse module, effectively avoiding information waste; regularization loss constrains the smoothness of the deformation field, preventing excessive distortion and folding. Joint training of these three loss functions enables the model to achieve better anatomical alignment performance in complex nonlinear deformation scenarios, and significantly reduces target registration error compared to traditional decoupling registration methods on public datasets.
[0083] Another aspect of the present invention provides a multimodal medical image registration system based on feature decoupling and reuse, the system comprising a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute the multimodal medical image registration method based on feature decoupling and reuse.
[0084] Another aspect of the present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the aforementioned multimodal medical image registration method based on feature decoupling and reuse.
[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0086] In summary, this invention separates structural and apparent features using a hierarchical feature decoupling device. It then utilizes a block-based cross-attention mechanism in the modal feature reuse module to effectively extract geometrically relevant information from the apparent features. This information is then fused with the structural features to jointly predict the primary and secondary deformation fields, thereby achieving secondary mining and utilization of potential structural information from "discarded" features. This method not only avoids wasting useful information but also significantly improves registration accuracy and robustness under complex nonlinear deformations through layer-by-layer refinement of the multi-level deformation fields, exhibiting higher anatomical alignment performance compared to traditional decoupling registration methods.
[0087] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A multimodal medical image registration method based on feature decoupling and reuse, characterized in that, include: S1: Obtain a multimodal medical image dataset, wherein the multimodal medical image dataset includes image pairs consisting of a first modal medical image and a second modal medical image, wherein the first modal medical image is a fixed image and the second modal medical image is a moving image; S2: Utilize a weight-sharing encoder network to extract multi-level hierarchical features from both moving and stationary images; S3: For both moving and stationary images, construct a hierarchical feature decoupler, a modal feature reuse module, and a deformation field prediction module for the hierarchical features at each level. S4: After applying the deformation field of the previous level to the hierarchical features of the moving image at the current level, it is input into the hierarchical feature decoupler corresponding to the current level, along with the hierarchical features of the fixed image at the current level, to separate the structural features of the fixed image, the structural features of the moving image, the appearance features of the fixed image, and the appearance features of the moving image at the current level. S5: Input the fixed image structure features, moving image structure features, fixed image appearance features, and moving image appearance features of the current level into the modal feature reuse module corresponding to the current level, and extract the structural association features and modal association features through the block cross attention mechanism; S6: Input the structural correlation features and modal correlation features into the deformation field prediction module corresponding to the current level, respectively, to predict the main deformation field and auxiliary deformation field of the current level; add the features of the main deformation field and auxiliary deformation field to obtain the initial deformation field of the current level; S7: Apply the deformation field of the previous level to the initial deformation field of the current level to obtain the deformation field of the current level; S8: Use the deformation field of the last layer as the final deformation field, and perform spatial transformation on the moving image through the final deformation field to obtain the registered moving image.
2. The multimodal medical image registration method based on feature decoupling and reuse according to claim 1, characterized in that, The weight-sharing encoder network includes two feature extraction branches with shared weights. Each feature extraction branch includes four cascaded convolutional blocks. Each convolutional block consists of cascaded convolutional layers and a ReLU activation function. The output features of all convolutional blocks in the two feature extraction branches are extracted and arranged in ascending order of scale to obtain a hierarchical feature set of the moving image. and the hierarchical feature set of a fixed image ;feature and Represents moving and stationary images. Hierarchical features at each level, .
3. The multimodal medical image registration method based on feature decoupling and reuse according to claim 2, characterized in that, Step S4 includes: S41: The deformation field of the previous level The displacement amplitude is magnified by 2 times, and then the deformation field is obtained by trilinear interpolation upsampling. In the first layer, there is no upper-layer deformation field, so it is directly taken as... Zero field; S42: Based on the deformation field Using spatial transformation networks Hierarchical features of the current layer of the moving image Features are obtained by deformation Then, the deformed hierarchical features Hierarchical features of the current layer fixed image The input hierarchical feature decoupler separates the fixed image structure features of the current level. Moving image structural features Fixed image appearance features and moving image appearance features .
4. The multimodal medical image registration method based on feature decoupling and reuse according to claim 3, characterized in that, The hierarchical feature decoupler includes: a first AMC module, a second AMC module, an AGS module, and four convolutional blocks; each convolutional block includes a cascaded convolutional layer, an IN normalization layer, and a ReLU activation function; the feature processing of the hierarchical feature decoupler includes: S401: Let the two input features of the hierarchical feature decoupler be... and , , , ; S402: Features and characteristics The first and second AMC modules are respectively input for context feature extraction to obtain features. and characteristics ; S403: Features and characteristics The AGS module is used to extract the structural features of modal sharing, resulting in the feature... and characteristics ; S404: Features and characteristics The image structure features are obtained by processing them through two convolutional blocks respectively. and moving image structural features ; S405: Features and characteristics After feature subtraction, the image is further processed through a convolutional block to obtain fixed image appearance features. ; S406: Features and characteristics After feature subtraction, the image is further processed through a convolutional block to obtain the apparent features of the moving image. .
5. The multimodal medical image registration method based on feature decoupling and reuse according to claim 4, characterized in that, The first AMC module and the second AMC module have the same network structure, both including convolutional layers and multiple layers with different hole ratios. convolution; The context feature extraction process in the first and second AMC modules includes: using multiple features with different hole rates Convolution extracts multi-scale contextual information from the input features, resulting in contextual features at multiple scales. The features are obtained by concatenating contextual features from all scales and then processing them through a convolutional layer. and characteristics .
6. The multimodal medical image registration method based on feature decoupling and reuse according to claim 4, characterized in that, The AGS module includes three convolutional processing layers, each of which comprises a cascaded convolutional layer and an IN normalization layer; the feature processing procedure of the AGS module includes: For input features and characteristics Features are obtained by processing through two convolutional layers respectively. and characteristics ; Features and characteristics After concatenation, the data is fed into a convolutional processing layer to obtain the features. ; Features The feature is obtained by uniformly splitting the channel along its dimension into two parts. and characteristics : in, Indicates features It is evenly divided into two features along the channel dimension; Based on characteristics and characteristics The relative weight matrix of features is calculated using Softmax normalization. According to characteristics and characteristics The relative weight matrix of features is calculated using Softmax normalization. : , ; ; ; ; ;in, Represents an exponential function; Relative weight matrix of features and characteristics The output features are obtained by performing element-wise multiplication. ; Relative weight matrix of features and characteristics The output features are obtained by performing element-wise multiplication. .
7. The multimodal medical image registration method based on feature decoupling and reuse according to claim 1, characterized in that, Step S5 includes: S51: Fix image structural features and moving image structural features By adding the features together, we obtain the fused structural features. ; S52: Integrating structural features Fixed image appearance features and moving image appearance features Each feature is divided into multiple non-overlapping feature blocks using the same block partitioning method; S53: Map the feature blocks with fused structural features to Q vectors through a linear layer, and map the image blocks with fixed image appearance features to first K vectors and first V vectors respectively through a linear layer; perform block-based cross-attention calculation based on Q vectors, first K vectors, and first V vectors to obtain first attention-weighted features; add the attention-weighted features to the feature blocks with fixed image appearance features to obtain enhanced first feature blocks; merge all enhanced first feature blocks to obtain modality association features; S54: Map the feature blocks of the appearance features of the moving image to the second K vector and the second V vector respectively through a linear layer; perform block cross-attention calculation based on the Q vector, the second K vector and the second V vector to obtain the second attention weighted feature; add the second attention weighted feature to the feature blocks of the appearance features of the moving image to obtain the enhanced second feature block; merge all the enhanced second feature blocks into blocks to obtain the structural association feature.
8. The multimodal medical image registration method based on feature decoupling and reuse according to claim 1, characterized in that, Step S7 includes: transferring the deformation field from the previous level... The displacement amplitude is magnified by 2 times, and then the deformation field is obtained by trilinear interpolation upsampling. According to the deformation field Using spatial transformation networks Initial deformation field at the current level Perform deformation to obtain the initial coarse deformation field of the current level; then combine the initial coarse deformation field of the current level with the initial deformation field of the current level. By summing the features, the deformation field of the current level is obtained. .
9. A multimodal medical image registration system based on feature decoupling and reuse, characterized in that, The system includes a memory and a processor; the memory is used to store an application program; the processor is used to run the application program and execute a multimodal medical image registration method based on feature decoupling and reuse as described in any one of claims 1 to 8.
10. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, implements a multimodal medical image registration method based on feature decoupling and reuse as described in any one of claims 1 to 8.