Method for multimodal medical image registration based on unsupervised deep learning and modal conversion

Abstract

The application relates to a multi-modal medical image registration method based on unsupervised deep learning and modal conversion, a modal conversion network of which is improved in image conversion quality from one mode to another mode through a generative adversarial network and added contrast loss, and the structure and details of the image are effectively reserved, a registration network of which is enhanced in sensitivity to image details through the introduction of contrast loss, mutual information loss and consistency loss, so that the accuracy and robustness are further improved in the registration process, and medical images with large modal differences can be accurately registered.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a multimodal medical image registration method based on unsupervised deep learning and modality transformation. Background Technology

[0002] With the rapid advancement of medical imaging technology, we can now observe the human body with unprecedented detail and accuracy, revealing detailed images of its internal structures and functions. This has greatly improved image quality and deepened our understanding of human anatomy and physiology. Medical imaging technology plays a crucial role in diagnosing various diseases, driving the development of the medical imaging discipline. For example, X-ray imaging can reveal the structure of bones and certain tissues, while techniques such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound, and PET (Positron Emission Tomography) provide more detailed views in different scenarios. In particular, structural imaging, with its high spatial resolution, reveals clear tissue structures, while functional imaging provides views of tissue metabolism, neural responses, and microscopic material changes, which are essential for understanding complex physiological processes. Therefore, in the process of using medical imaging to assist diagnosis, it is often necessary to fuse medical image information from different modalities to improve the accuracy of clinical diagnosis. However, due to the diverse sources of images from different modalities, they often have deviations in spatial alignment, making the application of image registration techniques particularly critical. Image registration technology ensures the feasibility of medical diagnosis and treatment planning based on complete and accurate information by precisely aligning images of different modalities in spatial location. In recent years, with the rise of deep learning, research on image registration using it has become a hot topic, especially the application of convolutional neural networks, which provides an effective way to overcome the limitations of traditional methods.

[0003] Multimodal medical image registration aims to align images of different modalities in space and anatomical locations to achieve precise matching of corresponding points, providing consistent, comprehensive, and accurate information. This is crucial for revealing disease progression and changes and has made significant progress in disease research, drug development, and surgical risk assessment. However, many registration tasks still rely on manual work by experts, which is not only time-consuming but also may yield inaccurate results and is difficult to handle large datasets. Therefore, developing fully automated multimodal medical image registration algorithms is essential. Traditional image registration strategies mainly revolve around physical models, such as viscous fluid, elastic, and differential homeomorphic models, or rely on feature matching based on salient features of the images (e.g., edges, corners). The core of these traditional methods lies in identifying and aligning image feature points to achieve spatial consistency between images. However, these methods are often limited by manually designed features and involve repeated feature extraction and matching for each pair of images to be registered, resulting in inefficiency. Deep learning techniques, especially Convolutional Neural Networks (CNNs), offer a new perspective on image registration. Convolutional neural networks (CNNs) can learn key features from images, overcoming the limitations of traditional methods that rely on manually generated features. More importantly, once a CNN model is trained, its parameters can be rapidly applied to new image registration tasks, significantly improving registration speed and efficiency and meeting the stringent real-time requirements of medical image registration. With the development of deep learning, especially CNNs, deep learning offers an effective solution for image registration, adaptively learning task-related features and dramatically improving registration speed to meet real-time processing needs.

[0004] However, many problems and challenges remain in the field of deep learning-driven multimodal medical image registration. Due to the heterogeneity of multimodal data, images produced by different imaging techniques exhibit significant differences in quality, contrast, and detail, thus increasing the complexity of registration. Compared to unimodal medical image registration, multimodal medical image registration lacks a universal similarity metric. Traditional intensity-based similarity metrics may not be applicable between different modalities because their intensity distributions may be completely different across modalities. This necessitates the development of new similarity measurement methods to adapt to multimodal scenarios. Furthermore, multimodal image registration typically requires processing large amounts of data and complex models, which can lead to high computational costs and significant time consumption, especially in clinical applications requiring real-time or near-real-time registration. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes an unsupervised deep learning-based medical image registration model based on modality transformation. The modality transformation network in the model significantly improves the image conversion quality from one modality to another through an improved generative adversarial network and an innovative contrast loss function, ensuring effective preservation of image structure and details. The registration network, by introducing a contrast loss, enhances its sensitivity to image details, thereby further improving accuracy and robustness during the registration process. Furthermore, this solves the problem of significant differences in quality, contrast, and detail between images produced by different imaging techniques in the field of multimodal medical image registration, enabling accurate and rapid registration of image pairs.

[0006] A multimodal medical image registration method based on unsupervised deep learning and modality transformation includes the following steps:

[0007] S1, preprocessing medical images, slicing three-dimensional medical images, normalizing gray values, expanding edges, performing affine transformation for coarse registration, and obtaining floating and fixed images;

[0008] S2, construct a modality conversion network, input the floating image into the modality conversion network to generate an image with a fixed image modality, i.e., a converted image. The modality conversion network is jointly trained by the contrast loss between the fixed image and the converted image and the adversarial loss between the floating image and the converted image.

[0009] S3, Construct a registration network, input the fixed image and the floating image into the registration network, and predict the deformation field of the transformation from the floating image to the fixed image;

[0010] S4. Using the spatial transformation network and the obtained deformation field, the floating image and the transformed image are respectively subjected to distortion transformation to obtain their respective registered images;

[0011] S5, calculate the mutual information loss and contrast loss between the image registered by the floating image and the fixed image, then calculate the similarity loss between the image registered by the transformed image and the fixed image, as well as the deformation field smoothing constraint loss. Train the registration network by combining the four losses, and repeat steps S2-S5 until the training is completed and then proceed to step S6.

[0012] S6. Repeat steps S2-S5 on the floating image and the fixed image to be registered to obtain the final registered image.

[0013] Furthermore, the specific process of step S1 is as follows: the three-dimensional medical image is sliced ​​and the slice data is standardized to map the gray value range of the data to between 0 and 255. The image size is adjusted to a uniform size and then saved as a PNG image. One of the processed modal images is selected as the floating image M and the other modal image is selected as the fixed image F. Then, the obtained fixed image and floating image are coarsely registered using affine transformation.

[0014] Furthermore, the specific process of step S2 is as follows: the modality conversion network is based on an improved generative adversarial network architecture, including a generator, a discriminator, and a feature extraction module for calculating the contrastive loss. The generator adopts an encoder-decoder structure, and the feature extraction module uses a pre-trained convolutional neural network of ResNet to extract features from the source image and the target image.

[0015] The overall loss function of the mode transfer network is:

[0016] L G =L PatchNCE +L D

[0017] Where L D For the adversarial loss of the discriminator, L PatchNCE Contrast loss in mode transfer networks:

[0018]

[0019] Where f is a neural network used to encode the input image into feature vectors, T is the image after M is transformed by the modality conversion network; sim(f(M), f(T)) is the similarity between feature vectors; N is the number of negative samples; τ is a temperature parameter that controls the smoothness of the similarity distribution.

[0020] Furthermore, the specific process of step S3 is as follows: a registration network is constructed using the VoxelMorph library, and an encoder-decoder architecture is adopted. The encoder is responsible for extracting image features, and the decoder is responsible for reconstructing the dense deformation field based on the features extracted by the encoder. The input of the registration network is a floating image and a fixed image, and the output is the deformation field.

[0021] Furthermore, the specific process of step S4 is as follows: Deformation is performed on the floating image and the transformed image through a spatial transformation network. The floating image and the transformed image are input into the spatial transformation network respectively, and a distortion transformation is performed using the obtained deformation field.

[0022]

[0023]

[0024] M′ and T′ are images of M and T after registration using the deformation field.

[0025] Furthermore, the specific process of step S5 is as follows: calculate various losses between the registered image and the fixed image, including mutual information loss, contrast loss, similarity loss, and deformation field smoothing constraint loss.

[0026] The mutual information loss between the image registered with the floating image and the fixed image is calculated as follows:

[0027]

[0028] Where, p F，M′ (f, m′) denotes the joint probability density function, p F (f) and p M′ (m′) represent the edge probability density functions of the fixed image and the registered image, respectively;

[0029] The contrast loss of the registration network is:

[0030]

[0031] The similarity loss between the converted image and the registered image and the fixed image is:

[0032]

[0033] The deformation field smoothing constraint loss is:

[0034]

[0035] Where (·) represents the calculation of the gradient.

[0036] The above steps together constitute the loss function in the image registration process. By minimizing these loss functions, we can obtain the optimal registration result. The specific process of step S6 is as follows: the floating image and the obtained optimal deformation field are input into the spatial transformation network through spatial input, and the obtained deformation field is used to perform a distortion transformation on them to obtain the final registered image.

[0037] Compared with the prior art, the beneficial effects of the present invention are:

[0038] 1) Contrastive learning enhancement for modality transformation: For modality transformation using generative adversarial networks, this invention maximizes the correlation between input and output image patches in the GAN-based image transformation network. This transformation network can focus on the common parts of the two images, such as shape, while also reducing the differences between the two images, such as reducing artifacts. At the same time, the transformed image can also retain more details of the source image.

[0039] 2) Joint Optimization of Mutual Information and Contrast Loss: To further improve registration accuracy, this invention proposes a joint loss function that combines mutual information loss and contrast loss. By calculating the contrast loss between the registered image and the fixed image pair, the local similarity between images is enhanced. Combining mutual information loss optimizes the overall information consistency of the images, ensuring global registration accuracy between different modalities, thereby achieving more accurate and reliable multimodal medical image registration. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of medical image preprocessing;

[0041] Figure 2 This is a network structure diagram of a mode conversion network;

[0042] Figure 3 This is a diagram of the registration network structure;

[0043] Figure 4 This is the registration result image.

[0044] Figure 5 This is a schematic diagram illustrating an implementation of a multimodal medical image registration method based on unsupervised deep learning and modality transformation. Detailed Implementation

[0045] In multimodal medical image registration, the inherent heterogeneity between multimodal images and the differences in image characteristics caused by different imaging mechanisms greatly increase the complexity of the registration process. Furthermore, traditional registration methods are often limited by the lack of effective similarity metrics, making the development of registration techniques suitable for multimodal environments an urgent research need. This invention proposes a multimodal medical image registration method based on unsupervised deep learning and modality transformation. The modality transformation network in the method significantly improves the image transformation quality from one modality to another through generative adversarial networks and the addition of contrast loss, ensuring the effective preservation of image structure and details. The registration network in the method enhances its sensitivity to image details by introducing contrast loss, mutual information loss, and consistency loss, thereby further improving accuracy and robustness during the registration process. The proposed method can accurately register medical images with significant modal differences.

[0046] The multimodal medical image registration method based on unsupervised deep learning and modality transformation of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] Example:

[0048] This example provides a multimodal medical image registration method for CT image registration to MRI image registration, which specifically includes the following processes:

[0049] S1, Preprocessing of CT and MRI medical images: such as Figure 1 As shown, the original NIfTI format image is sliced, and the slice data is standardized to map the data range to 0-255. The image size is adjusted to 224*224, and then saved as a PNG image. The CT modality image is selected as the floating image (M), and the MRI modality image is selected as the fixed image (F).

[0050] This invention pre-registers the floating image using the graphics processing tool ITK through affine transformation, which can effectively reduce the amount of deformation that needs to be processed in subsequent nonlinear registration.

[0051] S2, Construct the mode transfer network. The mode transfer network in this example has the following structure:

[0052] like Figure 2 As shown, the network's entry point is an initial convolutional layer responsible for mapping the input image to a high-dimensional feature space. Following this, a series of downsampling layers reduce the spatial dimensionality of the feature map through convolutional operations, increasing the feature depth. The core consists of nine ResNet blocks, each containing skip connections. These connections directly add the input to the output of the convolutional layers, helping to mitigate the vanishing gradient problem in deep networks and ensuring efficient information transfer. The final part of the network is an upsampling layer, which gradually increases the spatial dimensionality of the feature map through deconvolution or pixel random sampling techniques, ultimately mapping the features back to the desired number of output channels through an output convolutional layer. The entire network achieves non-linearity through batch normalization and the ReLU activation function.

[0053] The floating image is input into the modality conversion network to generate an image with a fixed image modality, namely the converted image T. The modality conversion network is jointly trained by the contrast loss between the fixed image and the converted image and the adversarial loss between the floating image and the converted image.

[0054] The overall loss function of the above mode conversion network is:

[0055] L G =L Patch +L D

[0056] Where L D For the adversarial loss of the discriminator, L PatchNCE Contrast loss in mode transfer networks:

[0057]

[0058] Here, f is a neural network used to encode the input image into feature vectors. sim(f(M), f(T)) is the similarity between feature vectors. N is the number of negative samples. τ is a temperature parameter that controls the "smoothness" of the similarity distribution.

[0059] The modality transformation network of this invention transforms an image from one modality to another while preserving important medical information of the original image, thereby reducing differences between different modalities.

[0060] S3. Construct the registration network. In this example, the registration network is built using the VoxelMorph library, such as... Figure 3 As shown, the entire registration network adopts an encoder-decoder architecture. The encoder is mainly responsible for extracting image features and consists of a series of convolutional layers with a feature channel configuration of [16, 32, 32, 64, 64, 64]. The decoder is responsible for reconstructing the dense deformation field based on the features extracted by the encoder, and its layer structure is [64, 64, 64, 32, 32, 32, 16]. In the decoder, upsampling layers are used to progressively recover the spatial dimensions of the feature map, while convolutional layers are used to fuse features and refine the predicted deformation field. Furthermore, skip connections are used to directly pass features from the encoder to the corresponding layers of the decoder, which helps to preserve and recover more spatial information, thereby improving the accuracy of registration. Fixed and floating images are input into the registration network to predict the deformation field transformed from the floating image to the fixed image. The input to the registration network is a floating image and a fixed image, and the output is the deformation field.

[0061] S4, the floating image and the transformed image are deformed using a spatial transformation network. The floating image and the transformed image are input into the spatial transformation network respectively, and the resulting deformation field is used to perform a distortion transformation on them. This process can be expressed by the following formula:

[0062]

[0063]

[0064] M′ and T′ are images of M and T after registration using the deformation field.

[0065] S5. In this step, we calculate various losses between the registered image and the fixed image, including mutual information loss, contrast loss, similarity loss, and deformation field smoothing constraint loss.

[0066] The above calculation of the mutual information loss between the registered floating image and the fixed image is as follows:

[0067]

[0068] Where, p F，M′(f, m′) denotes the joint probability density function, p F (f) and p M′ (m′) represent the edge probability density functions of the fixed image and the registered image, respectively.

[0069] The contrast loss of the registration network described above is:

[0070]

[0071] The similarity loss between the transformed image after registration and the fixed image described above is:

[0072]

[0073] The aforementioned deformation field smoothing constraint loss is:

[0074]

[0075] Where (·) represents the calculation of the gradient.

[0076] These steps together constitute the loss function in the image registration process. By minimizing these loss functions, we can obtain the optimal registration result.

[0077] Repeat steps S2-S5 until training is complete, then proceed to step S6.

[0078] S6, repeat steps S2-S5 on the floating CT image and the fixed MRI image to be registered to obtain the final registered image, such as... Figure 4 Shown.

[0079] This invention proposes a multimodal medical image registration method based on unsupervised deep learning and modality transformation, and explores a novel medical image registration framework. This framework employs generative adversarial networks and contrast loss to achieve image transformation between highly different modalities. The core of this invention lies in using a modality transformation network to convert an image from one modality to another while preserving important medical information of the original image, thereby reducing the differences between different modalities. To improve registration accuracy, this invention proposes a joint loss function combining contrast loss and mutual information loss for training the registration network.

[0080] The embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Various changes that can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal medical image registration method based on unsupervised deep learning and modality transformation, characterized in that, Includes the following steps: S1, preprocessing medical images, slicing three-dimensional medical images, normalizing gray values, expanding edges, performing affine transformation for coarse registration, and obtaining floating and fixed images; S2, construct a modality conversion network, input the floating image into the modality conversion network to generate an image with a fixed image modality, i.e., a converted image. The modality conversion network is jointly trained by the contrast loss between the fixed image and the converted image and the adversarial loss between the floating image and the converted image. S3, Construct a registration network, input the fixed image and the floating image into the registration network, and predict the deformation field of the transformation from the floating image to the fixed image; S4. Using the spatial transformation network and the obtained deformation field, the floating image and the transformed image are respectively subjected to distortion transformation to obtain their respective registered images; S5, calculate the mutual information loss and contrast loss between the image registered by the floating image and the fixed image, then calculate the similarity loss between the image registered by the transformed image and the fixed image, as well as the deformation field smoothing constraint loss. Train the registration network by combining the four losses, and repeat steps S2-S5 until the training is completed and then proceed to step S6. S6, repeat steps S2-S5 on the floating image and the fixed image to be registered to obtain the final registered image; The specific process of step S5 is as follows: calculate various losses between the registered image and the fixed image, including mutual information loss, contrast loss, similarity loss, and deformation field smoothing constraint loss. The mutual information loss between the image registered with the floating image and the fixed image is calculated as follows: ， in, Denotes the joint probability density function. and Let F represent the edge probability density functions of the fixed image and the registered image, respectively; F is the fixed image. For the registered image, f is a neural network used to encode the input image into a feature vector; The contrast loss of the registration network is: ， Where N is the number of negative samples. It is a temperature parameter that controls the smoothness of the similarity distribution; The similarity loss between the converted image and the registered image and the fixed image is: ， The deformation field smoothing constraint loss is: ， in This indicates the calculation of the gradient.

2. The multimodal medical image registration method based on unsupervised deep learning and modality transformation according to claim 1, characterized in that, The specific process of step S1 is as follows: the three-dimensional medical image is sliced ​​and the slice data is standardized to map the gray value range of the data to between 0 and 255. The image size is adjusted to a uniform size and then saved as a PNG image. One of the processed modal images is selected as the floating image M and the other modal image is selected as the fixed image F. Then, the obtained fixed image and floating image are coarsely registered using affine transformation.

3. The multimodal medical image registration method based on unsupervised deep learning and modality transformation according to claim 2, characterized in that, The specific process of step S2 is as follows: The modality conversion network is based on an improved generative adversarial network architecture, which includes a generator, a discriminator, and a feature extraction module for calculating the contrastive loss. The generator adopts an encoder-decoder structure, and the feature extraction module uses a pre-trained convolutional neural network of ResNet to extract features from the source image and the target image. The overall loss function of the mode transfer network is: ， in For the discriminator's adversarial loss, Contrast loss in mode transfer networks: ， Where f is a neural network used to encode the input image into a feature vector, and T is the image of M after being transformed by the modality conversion network; It represents the similarity between feature vectors; N is the number of negative samples; It is a temperature parameter that controls the smoothness of the similarity distribution.

4. The multimodal medical image registration method based on unsupervised deep learning and modality transformation according to claim 3, characterized in that, The specific process of step S3 is as follows: a registration network is constructed using the VoxelMorph library, and an encoder-decoder architecture is adopted. The encoder is responsible for extracting image features, and the decoder is responsible for reconstructing the dense deformation field based on the features extracted by the encoder. The input of the registration network is a floating image and a fixed image, and the output is the deformation field.

Images