Multi-modal image non-rigid registration method based on transfer learning
Through transfer learning and feature fusion of multi-layer perceptrons, combined with TPS and B-spline transformers, non-rigid registration of MRI and CBCT images is achieved, solving the problems of registration robustness and low accuracy in multimodal image fusion imaging, and improving the accuracy and stability of temporomandibular joint image registration.
Patent Information
- Application Number
- CN202510466854.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-15
- Publication Date
- 2025-08-08
AI Technical Summary
In the prior art, there are problems with low registration robustness and accuracy in the multimodal fusion imaging of MRI and CBCT images, especially in the key positions of the temporomandibular joint such as articular discs, condyles, etc., and there is a lack of comprehensive evaluation.
Using a multimodal image non-rigid registration method based on transfer learning, image features are extracted through the pre-trained ResNet-18 network, feature fusion is performed using a multi-layer perceptron MLP, and image registration is performed by combining the TPS converter and the B-spline converter to realize non-rigid registration of MRI and CBCT images.
It improves the accuracy and robustness of image registration, can better reflect soft and hard tissue characteristics, provides more reliable imaging support, and provides more efficient imaging support for the auxiliary diagnosis of temporomandibular joint disorders.
Smart Images
Figure CN120451228A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multimodal image non-rigid registration method based on transfer learning, and belongs to the technical field of image registration. Background Art
[0002] Temporomandibular disorders (TMD) are disorders of the temporomandibular joint or surrounding skeletal muscle system, with primary clinical manifestations including joint pain, joint clicking during movement, and mandibular movement disorders. Magnetic resonance imaging (MRI) and cone beam computer tomography (CBCT) are both widely used imaging modalities in the diagnosis and treatment of TMD. Their integration can overcome the limitations of single imaging modalities, leverage the strengths of both modalities, and achieve precise positioning, significantly improving the performance of radiotherapy, increasing the effective radiation dose, and reducing the risk of adverse reactions. However, MRI and CBCT images are typically acquired with different modalities and resolutions, resulting in image distortion and registration issues, which hinder accurate diagnosis and treatment planning.
[0003] In response to the above problems, non-rigid registration technology has become an effective method to solve the deformation and registration problems between images of different modalities. Non-rigid registration technology achieves alignment and matching between images of different modalities by deforming and matching local areas of the image. In the registration of temporomandibular joint images, traditional non-rigid registration methods usually require manual selection of feature points and calculation of feature descriptors, and then use these feature points for registration. However, manual selection of feature points is time-consuming and labor-intensive, and requires a high level of doctor experience. Therefore, an automated and efficient method is needed to achieve non-rigid registration of temporomandibular multimodal images.
[0004] Currently, preliminary studies have attempted to establish registration methods for MRI and cone-beam CT images. However, due to differences between the modalities during multimodal fusion imaging of the temporomandibular joint, registration robustness and accuracy are often low. Although some studies have suggested that multimodal image registration can produce images of the temporomandibular joint with complete soft and hard tissue structures, these studies are still in their infancy and primarily focus on PD-weighted images, closed-mouth views, and oblique sagittal planes. Key locations of the temporomandibular joint, such as the articular disc and condyle, are rarely addressed, lacking a comprehensive assessment.
[0005] Therefore, the multimodal image non-rigid registration method based on transfer learning proposed in the present invention provides a new solution to the problems of low registration robustness and accuracy in multimodal fusion imaging of the temporomandibular joint. Summary of the Invention
[0006] The purpose of the present invention is to provide a non-rigid registration method for multimodal images based on transfer learning to solve the problems in the prior art of insufficient utilization of multimodal information and the need to improve the accuracy, efficiency and robustness of registration.
[0007] The technical solution of the present invention is:
[0008] A non-rigid registration method for multimodal images based on transfer learning, comprising the following steps:
[0009] S1. Acquire an MRI image and a CBCT image of the temporomandibular joint, annotate the fossa, articular disc, and condyle of the MRI image of the temporomandibular joint, and annotate the fossa and condyle of the CBCT image. After reading and preprocessing the data, obtain paired preprocessed MRI images and preprocessed CBCT images as image samples to obtain an image dataset.
[0010] S2. Using transfer learning, the pre-trained ResNet-18 network is used to extract image features from image samples to obtain MRI image features and CBCT image features.
[0011] S3. Using a multi-layer perceptron (MLP) to fuse MRI image features and CBCT image features, obtaining a fused feature representation that includes information from all modalities, where the information from all modalities is feature information reflecting soft and hard tissue in both MRI and CBCT modalities, and obtaining predicted point coordinates.
[0012] S4, using the TPS transformer to register the predicted point coordinates, realizing non-rigid registration of the MRI image and the CBCT image, and obtaining the registered image M1;
[0013] S5. Perform B-spline transformation on the registered image M1 obtained in step S4 to register the image, and obtain the secondary registered CBCT and MRI image M2.
[0014] Furthermore, in step S1, an MRI image and a CBCT image of the temporomandibular joint are obtained, and after data reading and preprocessing, a pair of preprocessed MRI images and preprocessed CBCT images are obtained, specifically,
[0015] S11. Obtain a pair of MRI images and CBCT images corresponding to the same anatomical structure or region, wherein the MRI image format is DICOM, the CBCT image format is CTI, and the bit depth is a set value; read the MRI image and the CBCT image to obtain image data; and store the obtained image data in a tensor;
[0016] S12 . Preprocess the paired MRI images and CBCT images respectively to obtain paired preprocessed MRI images and preprocessed CBCT images.
[0017] Furthermore, in step S12, the paired MRI images and CBCT images are preprocessed separately. Specifically, the MRI images and CBCT images are first resized to set pixels using a bilinear interpolation method, and then histogram equalization is performed on the MRI images and CBCT images. Subsequently, the MRI images and CBCT images are normalized.
[0018] Furthermore, step S2 is specifically:
[0019] S21. Load the pre-trained ImageNet-based ResNet-18 network, freeze the convolutional layer weights of the ResNet-18 network, and perform transfer training only on the fully connected layers of the ResNet-18 network to obtain the transfer-trained ResNet-18 network for the TMJ multimodal task.
[0020] S22. Input the preprocessed MRI and CBCT images into the transfer-trained ResNet-18 network and extract feature representations through forward propagation.
[0021] Furthermore, in step S3, the multilayer perceptron model MLP includes an input layer, a hidden layer, and an output layer. The input layer is used to input the spliced features obtained by splicing the MRI image features and the CBCT image features into the hidden layer;
[0022] Hidden layer: performs nonlinear transformation and fusion to obtain the fused feature representation;
[0023] Output layer: used to output the fused feature representation.
[0024] Furthermore, step S4 is specifically:
[0025] S41, taking the predicted point coordinates obtained in step S3 as input, setting the predicted point coordinates as the source point coordinates of the TPS converter, and setting the point coordinates in the reference image as the target point coordinates of the TPS converter;
[0026] S42, setting parameters of the TPS converter, and optimizing the TPS converter to fit the conversion relationship from the source point to the target point;
[0027] S43 , using the image corresponding to the fused feature representation obtained in step S3 as the image to be registered M, transforming the image to be registered M to obtain an image to be registered M1 after the preprocessed CBCT image and the preprocessed MRI image are registered.
[0028] Furthermore, step S5 is specifically:
[0029] S51, converting the registered image M1 obtained in step S4 into a SimpleITK image format;
[0030] S52, initializing the B-spline transformation parameter space, setting the grid spacing, B-spline degree and automatically generated control point grid size so that the transformation range covers the entire image area;
[0031] S53, creating a B-spline registration instance, using the grid spacing, B-spline degree and automatically generated control point grid size set in step S52, and optimizing the B-spline transformation parameters, i.e., the displacement vectors of the control points, based on the registered image M1 and the reference image;
[0032] S54 , performing the registration operation, iterating until the maximum number of iterations or the change in the energy function is less than a set threshold, and outputting the image M2 after secondary registration.
[0033] The beneficial effects of the present invention are: compared with the existing technology, this multimodal image non-rigid registration method based on transfer learning uses a large-scale image data set to perform transfer training on a pre-trained deep learning model through transfer learning, which can achieve richer feature representation of images, improve the generalization ability of the model, and improve the accuracy and robustness of image registration, thereby providing more reliable imaging support for the auxiliary diagnosis of temporomandibular joint disorder. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 3 is a flow chart of a multimodal image non-rigid registration method based on transfer learning according to an embodiment of the present invention. DETAILED DESCRIPTION
[0035] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0036] The embodiment provides a multimodal image non-rigid registration method based on transfer learning, such as Figure 1 , including the following steps,
[0037] S1. Obtain MRI images and CBCT images of the temporomandibular joint, annotate the fossa, articular disc, and condyle on the MRI image of the temporomandibular joint, and annotate the fossa and condyle on the CBCT image. After reading and preprocessing the data, obtain pairs of preprocessed MRI images and preprocessed CBCT images as image samples to obtain an image dataset.
[0038] In step S1, MRI images and CBCT images of the temporomandibular joint are obtained, ensuring that the images contain annotations of key anatomical structures such as the articular disc and condyle. After reading the data and preprocessing, a pair of preprocessed MRI images and preprocessed CBCT images are obtained, specifically,
[0039] S11. Obtain a pair of MRI images and CBCT images corresponding to the same anatomical structure or region, wherein the MRI image format is DICOM, the CBCT image format is CTI, and the bit depth is a set value, such as 8 bits. Read the MRI image and the CBCT image to obtain image data, and store the obtained image data in a tensor.
[0040] S12 . Preprocess the paired MRI images and CBCT images respectively to obtain paired preprocessed MRI images and preprocessed CBCT images.
[0041] In step S12, the paired MRI and CBCT images are preprocessed separately. Specifically, the MRI and CBCT images are first resized to a set pixel size, such as 1024x1024 pixels, using bilinear interpolation. Histogram equalization is then performed on the MRI and CBCT images to enhance contrast and detail. The MRI and CBCT images are then normalized to ensure they have zero mean and unit standard deviation.
[0042] S2. Utilize transfer learning and use the pre-trained ResNet-18 network to extract image features from image samples to obtain MRI image features and CBCT image features.
[0043] S21. Load the pre-trained ImageNet-based ResNet-18 network, freeze the convolutional layer weights of the ResNet-18 network, and perform transfer training only on the fully connected layers of the ResNet-18 network to obtain the transfer-trained ResNet-18 network for the TMJ multimodal task.
[0044] S22. Input the preprocessed MRI and CBCT images into the transfer-trained ResNet-18 network and extract feature representations through forward propagation.
[0045] The ResNet-18 model is a deep residual network architecture that employs the concept of residual learning. It addresses degradation in deep neural network training by introducing residual connections, which allow the network to skip a layer during information flow, enabling it to better learn features. ResNet-18 contains 18 convolutional and fully connected layers, including four larger convolutional blocks. Each convolutional block consists of two convolutional layers and a residual connection. Batch normalization and activation functions are used for feature extraction and nonlinear activation. Finally, classification is performed through global average pooling and a fully connected layer that includes the number of categories. Using a pre-trained ResNet-18 to capture the rich and complex features of the input image can accelerate network convergence.
[0046] In step S2, transfer learning technology can use the knowledge and experience that have been learned to improve the learning performance on a new task.
[0047] S3. Use a multi-layer perceptron (MLP) to fuse MRI image features and CBCT image features to obtain a fused feature representation. The fused feature representation contains information from all modalities, where the information from all modalities is the feature information reflecting soft tissue and hard tissue in the MRI and CBCT modalities, and the predicted point coordinates are obtained.
[0048] In step S3, the multi-layer perceptron model MLP includes an input layer, a hidden layer, and an output layer.
[0049] Input layer: used to input the spliced features of MRI image features and CBCT image features into the hidden layer;
[0050] Hidden layer: performs nonlinear transformation and fusion to obtain the fused feature representation;
[0051] Output layer: used to output the fused feature representation.
[0052] In step S3, the feature vectors of the input MRI and CBCT images are defined, with a feature dimension of 100. A multilayer perceptron (MLP) model is defined to fuse the features of the MRI and CBCT images. The model consists of an input layer, a hidden layer, and an output layer. The dimension of the input layer is the sum of the dimensions of the features of the two modalities. The input feature dimension is set to 100, the hidden layer size is set to 50, and the output feature dimension is set to 20. The feature fusion model is then initialized. The features of the MRI and CBCT images are concatenated and input into the multilayer perceptron model for feature fusion. Nonlinear transformation and fusion are performed through a multilayer neural network to obtain a fused feature representation. A mean squared error loss function is selected and the loss is calculated. The model parameters are back-propagated and optimized using the Adam optimizer to update the model parameters. The predicted coordinates of the points are calculated and output. The fused feature representation contains information from all modalities, including soft and hard tissue feature information from the MRI and CBCT modalities. MRI image features primarily represent soft tissue features and can reflect soft tissue structures (such as dental pulp and articular disc). CBCT image features are mainly hard tissue features, which can reflect bone structures (such as teeth and jaws). The fused feature representation contains information from two modalities, which can comprehensively reflect the characteristics of soft and hard tissues, and contribute to more accurate image registration.
[0053] S4. Use the TPS transformer to register the predicted point coordinates to achieve non-rigid registration of the MRI image and the CBCT image, and obtain the registered image M1.
[0054] S41. Use the predicted point coordinates obtained in step S3 as input, set the predicted point coordinates as the source point coordinates of the TPS converter, and set the point coordinates in the reference image as the target point coordinates of the TPS converter; wherein the reference image refers to the standard image for aligning the target image during the registration process.
[0055] S42, setting parameters of the TPS converter, and optimizing the TPS converter to fit the conversion relationship from the source point to the target point;
[0056] S43 , using the image corresponding to the fused feature representation obtained in step S3 as the image to be registered M, transforming the image to be registered M to obtain an image to be registered M1 after the preprocessed CBCT image and the preprocessed MRI image are registered.
[0057] S5. Perform B-spline transformation on the registered image M1 obtained in step S4 to register the image, and obtain the secondary registered CBCT and MRI image M2.
[0058] S51, converting the registered image M1 obtained in step S4 into a SimpleITK image format;
[0059] S52, initialize the B-spline transformation parameter space, set the grid spacing to 2 times the pixel spacing, the B-spline degree to 3, and automatically calculate the control point grid size according to the image size to ensure that the transformation range covers the entire image area.
[0060] S53, creating a B-spline registration instance, using the grid spacing, B-spline degree and automatically generated grid size set in step S52, and optimizing the B-spline transformation parameters, i.e., the displacement vectors of the control points, based on the registered image M1 and the reference image;
[0061] In step S53, the B-spline transformation parameters are optimized based on the registered image M1 and the reference image. Specifically, based on the initialized parameter space configuration parameters, the grid spacing is set to twice the pixel spacing, the regularization weight is set to 0.1 to balance deformation smoothness and registration accuracy, the metric is set to Mattes mutual information, and the L-BFGS optimizer is used (maximum number of iterations 200, gradient convergence threshold 1e-5); at the same time, a random sampling strategy covering the key anatomical areas of the image is adopted (sampling ratio 30%) to accelerate the optimization.
[0062] S54 , performing the registration operation, iterating until the maximum number of iterations or the change in the energy function is less than a set threshold, and outputting the image M2 after secondary registration.
[0063] Through the above steps S51-S55, non-rigid registration of multimodal images can be achieved, so that images of different modalities can be aligned in the same space. By optimizing the B-spline transformation parameters and sampling strategy, the registration accuracy is improved.
[0064] S6. Extraction of parameters and registration graph: Obtain the image M2 after secondary registration and save it in a variable. Obtain the transformation parameters after registration and save them in a variable.
[0065] S7. Evaluation and Optimization: Evaluate the secondary registered image M2, for example by calculating the registration error and comparing image similarities, to assess the registration quality. Based on the evaluation results, adjust the B-spline transformation parameters or optimize the SGD algorithm to improve the registration accuracy and stability. Based on the optimization results, conduct subsequent medical image analysis or other related research. For example, analyze the morphology, structure, or function of key locations in the temporomandibular joint to provide data support for subsequent diagnosis and treatment.
[0066] This transfer learning-based multimodal image non-rigid registration method, specifically for the registration of MRI and CBCT images of the temporomandibular joint (TMJ), leverages multimodal information to improve registration accuracy and robustness. Transfer learning helps the model leverage the knowledge and experience learned from the original task when handling new tasks, accelerating its convergence. In the field of medical imaging, transfer learning can better utilize already labeled data, saving the time and resources required to annotate new data and improving the accuracy and efficiency of non-rigid registration.
[0067] This transfer learning-based multimodal image non-rigid registration method utilizes the key locations of the temporomandibular joint (TMJ) to achieve more accurate non-rigid registration. This helps improve the accuracy and stability of medical image processing, providing doctors with more reliable image data for diagnosis and treatment decisions. The key locations of the TMJ provide a more meaningful reference point in medical images, helping to reduce image alignment issues caused by non-rigid deformations. This allows for more accurate registration, thereby improving the effectiveness and reliability of image analysis.
[0068] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.
Claims
1. A multimodal image non-rigid registration method based on transfer learning, characterized by: The following steps are included: S1. Acquire an MRI image and a CBCT image of the temporomandibular joint, annotate the fossa, articular disc, and condyle of the MRI image of the temporomandibular joint, and annotate the fossa and condyle of the CBCT image. After reading and preprocessing the data, obtain paired preprocessed MRI images and preprocessed CBCT images as image samples to obtain an image dataset. S2. Using transfer learning, the pre-trained ResNet-18 network is used to extract image features from image samples to obtain MRI image features and CBCT image features. S3. Using a multi-layer perceptron (MLP) to fuse MRI image features and CBCT image features, obtaining a fused feature representation that includes information from all modalities, where the information from all modalities is feature information reflecting soft and hard tissue in both MRI and CBCT modalities, and obtaining predicted point coordinates. S4, using the TPS transformer to register the predicted point coordinates, realizing non-rigid registration of the MRI image and the CBCT image, and obtaining the registered image M1; S5. Perform B-spline transformation on the registered image M1 obtained in step S4 to register the image, and obtain the secondary registered CBCT and MRI image M2.
2. The multimodal image non-rigid registration method based on transfer learning according to claim 1, characterized in that: In step S1, an MRI image and a CBCT image of the temporomandibular joint are acquired, and after data reading and preprocessing, a pair of preprocessed MRI images and preprocessed CBCT images are obtained, specifically, S11. Obtain a pair of MRI images and CBCT images corresponding to the same anatomical structure or region, wherein the MRI image format is DICOM, the CBCT image format is CTI, and the bit depth is a set value; read the MRI image and the CBCT image to obtain image data; and store the obtained image data in a tensor; S12 . Preprocess the paired MRI images and CBCT images respectively to obtain paired preprocessed MRI images and preprocessed CBCT images.
3. The multimodal image non-rigid registration method based on transfer learning according to claim 2, characterized in that: In step S12, the paired MRI images and CBCT images are preprocessed separately. Specifically, the MRI images and CBCT images are first resized to set pixels using a bilinear interpolation method, and then histogram equalization is performed on the MRI images and CBCT images. Subsequently, the MRI images and CBCT images are normalized.
4. The multimodal image non-rigid registration method based on transfer learning according to claim 1, characterized in that: Step S2 is specifically: S21. Load the pre-trained ImageNet-based ResNet-18 network, freeze the convolutional layer weights of the ResNet-18 network, and perform transfer training only on the fully connected layers of the ResNet-18 network to obtain the transfer-trained ResNet-18 network for the TMJ multimodal task. S22. Input the preprocessed MRI and CBCT images into the transfer-trained ResNet-18 network and extract feature representations through forward propagation.
5. The method for non-rigid registration of multimodal images based on transfer learning according to any one of claims 1 to 3, characterized in that: In step S3, the multi-layer perceptron model MLP includes an input layer, a hidden layer, and an output layer. Input layer: used to input the spliced features of MRI image features and CBCT image features into the hidden layer; Hidden layer: performs nonlinear transformation and fusion to obtain the fused feature representation; Output layer: used to output the fused feature representation.
6. The method for non-rigid registration of multimodal images based on transfer learning according to any one of claims 1 to 3, characterized in that: Step S4 is specifically: S41, taking the predicted point coordinates obtained in step S3 as input, setting the predicted point coordinates as the source point coordinates of the TPS converter, and setting the point coordinates in the reference image as the target point coordinates of the TPS converter; S42, setting parameters of the TPS converter, and optimizing the TPS converter to fit the conversion relationship from the source point to the target point; S43 , using the image corresponding to the fused feature representation obtained in step S3 as the image to be registered M, transforming the image to be registered M to obtain an image to be registered M1 after the preprocessed CBCT image and the preprocessed MRI image are registered.
7. The method for non-rigid registration of multimodal images based on transfer learning according to any one of claims 1 to 3, characterized in that: Step S5 is specifically: S51, converting the registered image M1 obtained in step S4 into a SimpleITK image format; S52, initializing the B-spline transformation parameter space, setting the grid spacing, B-spline degree and automatically generated control point grid size so that the transformation range covers the entire image area; S53, creating a B-spline registration instance, using the grid spacing, B-spline degree and automatically generated control point grid size set in step S52, and optimizing the B-spline transformation parameters, i.e., the displacement vectors of the control points, based on the registered image M1 and the reference image; S54 , performing the registration operation, iterating until the maximum number of iterations or the change in the energy function is less than a set threshold, and outputting the image M2 after secondary registration.