Multi-modal image registration model and method based on deep learning

Abstract

The invention relates to the technical field of image registration, and discloses a multi-modal image registration model and method based on deep learning, and the model comprises a fine-grained feature extraction module, a registration network, and a real-time guidance optimization module. The fine-grained feature extraction module is of a multi-layer encoder-decoder structure, and an extrusion and excitation network SE-Net is introduced into each layer of encoder; and the real-time guidance optimization module comprises a space conversion network. The method solves the problem that the prior art is not suitable for complex scenes, and has the characteristics of being independent of a CPU (Central Processing Unit) and suitable for real-time application.

Description

Technical Field

[0001] The present invention relates to the technical field of image registration, and more specifically, to a multimodal image registration model and method based on deep learning. Background Art

[0002] Medical image registration is an important part of various medical image analysis applications. Image registration input usually includes a pair of floating images and fixed images. The transformation model is used to integrate different image sets into a single coordinate system with the same information, providing doctors with more comprehensive and accurate patient anatomical structure and lesion information, assisting in accurate lesion localization, preoperative evaluation and surgical navigation. Because medical images of different modalities contain complementary information about anatomical structures and lesion tissues. Therefore, in clinical practice, doctors need to fuse image registrations of different modalities and make accurate diagnosis and treatment decisions with the help of the rich information of multimodal images.

[0003] Existing medical image registration methods can be divided into linear registration (including rigid registration, affine registration, etc.) and nonlinear registration (deformable registration). Rigid and affine transformations achieve deformation by learning the transformed 2D / 3D matrix. Nonlinear transformations need to be represented by 3D matrices (for 2D deformable registration) or 4D matrices (for 3D registration), also known as deformation fields or displacement vector fields (DVF). In recent years, due to the development of neural network technology, image registration methods based on deep learning have become increasingly popular, mainly including end-to-end learning, feature matching, autoencoders, generative adversarial networks (GANs), convolutional long short-term memory networks (ConvLSTMs), and attention mechanisms. They mainly extract image features through deep learning models such as convolutional neural networks, and use designed loss functions such as mutual information or mean square error to measure the registration quality. The transformation model parameters are adjusted through optimization algorithms such as gradient descent to achieve accurate alignment between the source image and the target image. This process involves data preprocessing, model training, registration application, and evaluation of registration accuracy, aiming to improve the accuracy and efficiency of medical image analysis and provide support for clinical diagnosis and treatment.

[0004] However, the existing technology is not suitable for complex scenes. Therefore, how to invent an image registration model and method suitable for complex scenes is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0005] In order to solve the problem that the prior art is not suitable for complex scenes, the present invention provides a multimodal image registration model and method based on deep learning, which has the characteristics of being independent of CPU and suitable for real-time applications.

[0006] In order to achieve the above-mentioned purpose of the present invention, the technical scheme adopted is as follows: A multimodal image registration model based on deep learning includes a fine-grained feature extraction module, a registration network, and a real-time guidance optimization module; the fine-grained feature extraction module is a multi-layer encoder-decoder structure, and a squeeze and excitation network SE-Net is introduced in each layer of the encoder; the real-time guidance optimization module includes a spatial transformation network.

[0007] Preferably, the registration network is based on a transformation-based image registration model TransMorph; specifically, TransMorph with a layer of sliding window transformer block swin deleted is used.

[0008] Furthermore, specifically, the fine-grained feature extraction module specifically includes a 3-layer encoder-decoder structure.

[0009] Furthermore, in the fine-grained feature extraction module, the first layer encoder includes the first squeeze and excitation convolution block and the first maximum pooling layer; the second layer encoder includes the second squeeze and excitation convolution block and the second maximum pooling layer; the third layer encoder includes the third squeeze and excitation convolution block; the first layer decoder includes the first deconvolution block; the second layer decoder includes the second deconvolution block; and the third layer decoder includes 1 convolution block.

[0010] Furthermore, the first squeeze and excitation convolution block, the first maximum pooling layer, the second squeeze and excitation convolution block, the second maximum pooling layer, the third squeeze and excitation convolution block, the first deconvolution block, the second deconvolution block, and a convolution block are connected in sequence; the first squeeze and excitation block and the convolution block, the second squeeze and excitation block and the second deconvolution block are jump-connected.

[0011] Furthermore, the spatial transformation network includes a positioning network connected in sequence for generating transformation parameters according to an input image, a transformation application block for applying a corresponding transformation to the input image according to the generated parameters, and a sampler for performing image resampling.

[0012] A multimodal image registration method based on deep learning, based on the multimodal image registration model, includes the following specific steps: Get floating and fixed images; Segment the floating image and the fixed image to obtain a floating segmented image and a fixed segmented image; Inputting the floating image, the fixed image, the floating segmented image, and the fixed segmented image into a multimodal image registration model; Extracting features of floating images and fixed images through a fine-grained feature extraction module; The extracted features are input into the registration network for registration to obtain the predicted deformation field ; The predicted deformation field As a pseudo-label; , the fixed image and the floating segmentation map are input into the space transformation network to obtain the warped image; Apply the predicted deformation field to the segmentation map of the floating image and perform spatial network transformation to obtain the corresponding warped segmentation map; The total optimization function is calculated based on the warped segmentation map and iterative training is performed to obtain the final warped image.

[0013] Preferably, during iterative training, the Adam optimizer is used for iterative optimization to obtain the optimized deformation field .

[0014] Further, iterative training, the specific steps are: Compute fixed segmentation map and warping segmentation map The similarity index between them is used, and MSE is used to measure the predicted deformation field and optimized deformation field The difference is used to ensure the alignment between voxels and pixels, and the loss function is obtained:

[0015] in, yes The total prime number of represent The voxels in represent and The dice loss, Calculate the number and proportion of elements less than 0 in the Jacobian determinant of the deformation field. The higher the value, the rougher the deformation field and the more serious the folding distortion. Conversely, the smoother the deformation field. ; Representing volume space voxels in , and in and The same estimation method is used in the direction; , ; calculate , and based on The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm; According to the gradient information, the registration network is updated through the Adam optimizer to update , and conduct the next round of iterative training until the iteration indicators are completed.

[0016] Furthermore, during iterative training, the energy function to be minimized is also considered. , calculate the loss function :

[0017] Based on and The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm.

[0018] The beneficial effects of the present invention are as follows: The present invention discloses a multimodal image registration model based on deep learning, which uses a fine-grained feature extraction module to extract multi-scale multi-channel features and focuses on the unique features of the modality itself, while introducing a real-time guidance optimization module to optimize the output and reduce unnatural deformation. The present invention solves the problem that the prior art is not suitable for complex scenes, and has the characteristics of being independent of the CPU and suitable for real-time applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a structural diagram of a multimodal image registration model based on deep learning.

[0020] Figure 2 It is a schematic diagram of the structure of the fine-grained feature extraction module.

[0021] Figure 3 It is a flowchart of a multimodal image registration method based on deep learning.

[0022] Figure 4 It is a schematic diagram of the stage flow of a multimodal image registration method based on deep learning.

[0023] Figure 5 It is a schematic diagram of iterative optimization of the real-time guidance optimization module.

[0024] Figure 6 It is a schematic diagram of quantitative comparison of the registration results between different models. DETAILED DESCRIPTION

[0025] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] Example 1 like Figure 1 As shown, a multimodal image registration model based on deep learning includes a fine-grained feature extraction module, a registration network, and a real-time guidance optimization module; the fine-grained feature extraction module is a multi-layer encoder-decoder structure, and a squeeze and excitation network SE-Net is introduced in each layer of the encoder; the real-time guidance optimization module includes a spatial transformation network.

[0027] In a specific embodiment, the registration network is based on the transformation-based image registration model TransMorph; specifically, TransMorph is used in which a layer of sliding window transformer module swin transformer block is deleted to reduce memory overhead and adapt to data size, and the default parameter configuration recommended by TransMorph is used.

[0028] In a specific embodiment, specifically, the fine-grained feature extraction module specifically includes a 3-layer encoder-decoder structure.

[0029] In a specific embodiment, Figure 2 As shown, in the fine-grained feature extraction module, the first layer encoder includes the first squeeze and excitation convolution block and the first maximum pooling layer; the second layer encoder includes the second squeeze and excitation convolution block and the second maximum pooling layer; the third layer encoder includes the third squeeze and excitation convolution block; the first layer decoder includes the first deconvolution block; the second layer decoder includes the second deconvolution block; the third layer decoder includes 1 convolution block.

[0030] In a specific embodiment, the first squeeze and excitation convolution block, the first maximum pooling layer, the second squeeze and excitation convolution block, the second maximum pooling layer, the third squeeze and excitation convolution block, the first deconvolution block, the second deconvolution block, and a convolution block are connected in sequence; the first squeeze and excitation block and the convolution block, the second squeeze and excitation block and the second deconvolution block are jump-connected through a Concat operation.

[0031] In this embodiment, the first, second and third squeeze and excitation convolution blocks are composed of the cascaded first three-dimensional convolution layer, the first BN layer, the second three-dimensional convolution layer, the second BN layer, the LeakyReLU layer, and the SE block; the convolution block is composed of the cascaded first three-dimensional convolution layer, the first BN layer, the second three-dimensional convolution layer, the second BN layer, and the LeakyReLU layer; the first and second deconvolution blocks are composed of the cascaded three-dimensional deconvolution layer, the BN layer, and the LeakyReLU layer.

[0032] In a specific embodiment, the spatial transformation network includes a positioning network connected in sequence for generating transformation parameters according to an input image, a transformation application block for applying a corresponding transformation to the input image according to the generated parameters, and a sampler for performing image resampling.

[0033] In this embodiment, a fine-grained feature extractor FE based on convolutional neural network and 3D channel attention mechanism is designed. The extractor uses U-shaped structure and jump connection to fuse the different scale features of the modality itself, and introduces squeezing and excitation network in the downsampling stage of encoding to enhance feature learning ability. The feature extractor extracts the multi-channel and different scale features of the fusion modality itself and learns the correspondence between volumes in advance to obtain two fine-grained feature maps, which are then integrated and input into the registration learning network.

[0034] Example 2 like Figure 3 As shown, a multimodal image registration method based on deep learning, based on the multimodal image registration model, includes the following specific steps: Get floating and fixed images; Segment the floating image and the fixed image to obtain a floating segmented image and a fixed segmented image; Inputting the floating image, the fixed image, the floating segmented image, and the fixed segmented image into a multimodal image registration model; Extracting features of floating images and fixed images through a fine-grained feature extraction module; The extracted features are input into the registration network for registration to obtain the predicted deformation field ; The predicted deformation field As a pseudo-label; , the fixed image and the floating segmentation map are input into the space transformation network to obtain the warped image; Apply the predicted deformation field to the segmentation map of the floating image and perform spatial network transformation to obtain the corresponding warped segmentation map; The total optimization function is calculated based on the warped segmentation map and iterative training is performed to obtain the final warped image.

[0035] In a specific embodiment, Figure 5 As shown in the figure, during iterative training, the Adam optimizer is used to perform n iterations of optimization to obtain the optimized deformation field. .

[0036] In a specific embodiment, the iterative training comprises the following specific steps: Compute fixed segmentation map and warping segmentation map The similarity index between them is used, and MSE is used to measure the predicted deformation field and optimized deformation field The difference is used to ensure the alignment between voxels and pixels, and the loss function is obtained:

[0037] in, yes The total prime number of represent The voxels in represent and The dice loss, Calculate the number and proportion of elements less than 0 in the Jacobian determinant of the deformation field. The higher the value, the rougher the deformation field and the more serious the folding distortion. Conversely, the smoother the deformation field. ; Representing volume space voxels in , and in and The same estimation method is used in the direction; , ; calculate , and based on The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm; According to the gradient information, the registration network is updated through the Adam optimizer to update , and conduct the next round of iterative training until the iteration indicators are completed.

[0038] In a specific embodiment, during iterative training, the energy function to be minimized is also considered. , calculate the loss function :

[0039] Based on and The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm.

[0040] Previous learning-based registration frameworks usually directly add the fixed volume and the moving volume into one volume and input it into the registration network to directly capture the spatial correspondence between the two volumes. However, this does not fully consider the feature differences between different volumes, especially in multimodal registration work, where the image intensity distribution between modalities is very inconsistent. In this case, the direct input method into the network may not fully exert the registration performance.

[0041] In this embodiment, the multimodal image registration method based on deep learning is modified based on the previous single-modal registration method using deep learning, and the single-mode registration model is migrated to the multimodal registration work, and then a framework suitable for multimodal medical image registration is proposed. A feature extractor FE suitable for multimodality is designed, using a U-shaped structure and jump connection to learn features between different modalities, and introducing a channel attention mechanism to enhance feature learning. A dynamic optimization module is introduced, using the registered deformation field as a pseudo-gold standard to improve the current predicted deformation field.

[0042] In this embodiment, the registration method of the present invention does not rely on the real deformation field, nor does it directly use the deformation field generated by the registration network, but uses the predicted deformation field As a pseudo-label, the final deformation field is obtained by iterative optimization. It is worth noting that no additional parameters are introduced in the optimization stage, only a spatial transformer network is included, and the optimization stage is only performed in the training stage, which allows the trained model to skip the optimization process and directly perform inference. Adam is used as the optimizer to perform gradient backpropagation to update the optimized deformation field. This dynamic optimization strategy. Since the registration model generates a high-quality initial deformation field, it provides a better starting point for the optimizer, so that the optimized deformation field is further obtained after n iterations. At the same time, the optimized deformation field in turn enhances the supervision of the model, forming a positive feedback mechanism. The present invention not only saves computing costs due to a small number of optimization steps during the training process, but also the optimized deformation can also be the goal achieved by the ongoing training, providing more direct guidance for the adjustment of model parameters. In essence, real-time dynamic optimization provides continuous and gradually enhanced supervision, and its self-improvement characteristics provide continuous and purposeful guidance for model training.

[0043] Example 3 In this embodiment, Figure 6 As shown in Figure 1, four image registration baseline methods were used for comparative experiments on the SR-Reg dataset, including three multimodal image registration methods based on deep learning, VoxelMorph, VIT-V-Net, and TransMorph. The experimental results are shown in Table 1: Table 1

[0044] This deep learning-based multimodal image registration method improves the DSC index by 6%, 3.9% and 1.2% respectively compared with the existing learning-based registration models VoxelMorph, ViT-V-Net and TransMorph. In addition, qualitative experiments comparing different mainstream models show significant advantages in preventing the generation of excessive folding or sharp deformation fields, indicating that it can improve the registration performance while maintaining the smoothness and naturalness of the results.

[0045] In this embodiment, an ablation experiment is also performed on the module designed by the multimodal image registration method based on deep learning, and the results are shown in Table 2: Table 2

[0046] The results show that the registration accuracy is significantly improved after adding the fine-grained feature extraction module FE to the same model, and the smoothness of the registration result is significantly improved after introducing the real-time guidance optimization module Opt, which verifies the improvement of the registration effect by the FE module and the Opt module.

[0047] In summary, in the same multimodal registration scenario, the accuracy of registration and the smoothness of the registration results of this method are better than those of mainstream registration models represented by TransMorph and VoxelMorph. Compared with the latter, the OMMorph framework extends the work of OFG on single-modal registration to multimodality, and verifies its effectiveness through a series of experiments. In general, this method shows significant superiority in registration accuracy, the ability to prevent the generation of excessive folding or sharp deformation fields, and significantly outperforms traditional registration techniques in inference time.

[0048] It can be seen that the framework design of the present invention allows adjustment and application between different medical image modalities, and has good flexibility and scalability. The framework is not limited to the registration between CT and MR images. For medical images of other modalities and other parts, cross-modal registration and fusion can be achieved as long as the required segmentation map and corresponding modality image are provided. In addition, the invention can also be used in other medical fields that require high-precision image registration, such as surgical planning and postoperative evaluation.

[0049] Obviously, the above embodiments of the present invention are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A multimodal image registration model based on deep learning, characterized by: It includes a fine-grained feature extraction module, a registration network, and a real-time guidance optimization module; the fine-grained feature extraction module is a multi-layer encoder-decoder structure, and a squeeze and excitation network SE-Net is introduced in each layer of the encoder; the real-time guidance optimization module includes a space conversion network.

2. The multimodal image registration model based on deep learning according to claim 1, characterized in that: The registration network is based on the transformation-based image registration model TransMorph; specifically, TransMorph with one layer of sliding window transformer module swin transformer block deleted is used.

3. The multimodal image registration model based on deep learning according to claim 1, characterized in that: Specifically, the fine-grained feature extraction module includes a 3-layer encoder-decoder structure.

4. The multimodal image registration model based on deep learning according to claim 3, characterized in that: In the fine-grained feature extraction module, the first-layer encoder includes the first squeeze and excitation convolution block and the first maximum pooling layer; the second-layer encoder includes the second squeeze and excitation convolution block and the second maximum pooling layer; the third-layer encoder includes the third squeeze and excitation convolution block; the first-layer decoder includes the first deconvolution block; the second-layer decoder includes the second deconvolution block; and the third-layer decoder includes 1 convolution block.

5. The multimodal image registration model based on deep learning according to claim 4, characterized in that: The first squeeze and excitation convolution block, the first maximum pooling layer, the second squeeze and excitation convolution block, the second maximum pooling layer, the third squeeze and excitation convolution block, the first deconvolution block, the second deconvolution block, and one convolution block are connected in sequence; the first squeeze and excitation block and the convolution block, the second squeeze and excitation block and the second deconvolution block are jump-connected.

6. The multimodal image registration model based on deep learning according to claim 5, characterized in that: The spatial transformation network includes a positioning network for generating transformation parameters according to an input image, a transformation application block for applying a corresponding transformation to the input image according to the generated parameters, and a sampler for performing image resampling, which are connected in sequence.

7. A multimodal image registration method based on deep learning, characterized in that: Based on the multimodal image registration model according to any one of claims 1 to 6, the method comprises the following specific steps: Get floating and fixed images; Segment the floating image and the fixed image to obtain a floating segmented image and a fixed segmented image; Inputting the floating image, the fixed image, the floating segmented image, and the fixed segmented image into a multimodal image registration model; Extracting features of floating images and fixed images through a fine-grained feature extraction module; The extracted features are input into the registration network for registration to obtain the predicted deformation field ; The predicted deformation field As a pseudo-label; , the fixed image and the floating segmentation map are input into the space transformation network to obtain the warped image; Apply the predicted deformation field to the segmentation map of the floating image and perform spatial network transformation to obtain the corresponding warped segmentation map; The total optimization function is calculated based on the warped segmentation map and iterative training is performed to obtain the final warped image.

8. The multimodal image registration method based on deep learning according to claim 7, characterized in that: During iterative training, the Adam optimizer is used for iterative optimization to obtain the optimized deformation field .

9. The multimodal image registration method based on deep learning according to claim 8, characterized in that: Iterative training, the specific steps are: Compute fixed segmentation map and warping segmentation map The similarity index between them is used, and MSE is used to measure the predicted deformation field and optimized deformation field The difference is used to ensure the alignment between voxels and pixels, and the loss function is obtained: in, yes The total prime number of represent The voxels in represent and The dice loss, Calculate the number and proportion of elements less than 0 in the Jacobian determinant of the deformation field. The higher the value, the rougher the deformation field and the more serious the folding distortion. Conversely, the smoother the deformation field. ; Representing volume space voxels in , and in and The same estimation method is used in the direction; , ; calculate , and based on The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm; According to the gradient information, the registration network is updated through the Adam optimizer to update , and conduct the next round of iterative training until the iteration indicators are completed.

10. The multimodal image registration method based on deep learning according to claim 9, characterized in that: During iterative training, the energy function to be minimized is also considered , calculate the loss function : Based on and The gradient of the loss with respect to the deformation field is calculated through the back-propagation algorithm.

Images