A Medical Image Registration Method Based on Self-Supervised Multimodal Fusion

Abstract

The present invention relates to a medical image registration method based on self-supervised multimodal fusion. The method comprises the following steps: obtaining a pair of CT and MRI images of the same patient; employing an MRI image super-resolution reconstruction model trained with coronal and sagittal slices of the MRI image as supervisory signals to obtain a high-resolution MRI image; using the CT image as a fixed image and the MRI image as a moving image to be deformed, respectively, using the original MRI image and the CT image training model, as well as the high-resolution MRI image and CT image obtained by the reconstruction model to train a deformation field generation model to obtain a coarse-scale deformation field and a fine-scale deformation field, and superimposing the two deformation fields to obtain a hybrid deformation field for registering the patient's CT and MRI images. Compared with existing technologies, the present invention addresses the computational challenges associated with traditional non-rigid registration methods, improves the speed and efficiency of image registration, and is more suitable for clinical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image registration, and in particular to a medical image registration method based on self-supervised multimodal fusion. Background Art

[0002] In the diagnosis and treatment of head and neck tumors, it is crucial to accurately obtain the spatial distribution and tissue type of the tumor. Because tumors may invade both soft tissue and bone tissue at the same time, in complex cases, a single modality may not be able to fully display the full picture of the lesion, so a comprehensive evaluation requires the use of multiple imaging technologies. Computed tomography (CT) images have better imaging effects on bones and high-density tissues, while magnetic resonance imaging (MRI) performs better on soft tissues (such as muscles, brain, organs, etc.). However, for these two modal image data, doctors need to simultaneously view the invasion of the tumor on both images when making a diagnosis. Unaligned dual-modality images are often unable to be accurately aligned due to factors such as changes in patient position and differences in equipment imaging angles, resulting in doctors being unable to directly compare the features of the same anatomical structure. Because obtaining structurally aligned dual-modality images is particularly important for accurately locating lesions and surrounding key structures, the process of lesion diagnosis, evaluation, and treatment is optimized.

[0003] Image registration is the process of aligning two or more images to the same coordinate system, enabling accurate matching of corresponding structures within the images. The primary goal of this invention is to achieve bidirectional registration between CT and MRI images—registering CT images to MRI images, and vice versa. This bidirectional registration method can provide more comprehensive diagnostic information, helping doctors visualize bone and soft tissue images simultaneously from the same perspective, enabling accurate assessment of tumor spread and invasiveness.

[0004] Traditional image registration often uses non-rigid deformation-based image registration algorithms to obtain the deformation field. This involves complex optimization processes and large-scale deformation field solutions, consumes large amounts of computing resources, and runs slowly. When processing large, high-resolution images, the computation time is long, making it particularly inefficient in real-time applications or large-scale data processing scenarios. Furthermore, the performance of non-rigid deformation-based registration algorithms depends heavily on parameter settings, such as the smoothness of the deformation field and regularization weights. Improper parameter settings can lead to overfitting or underfitting, compromising the quality of the registration results.

[0005] MRI scans require a long time for data acquisition, usually tens of minutes or even longer. If you want to improve the resolution, you need to increase the scanning time to improve the signal-to-noise ratio. However, too long a scanning time is not only uncomfortable for the patient, but also easily causes artifacts due to unconscious movements (such as breathing, small movements), which in turn affects the image quality. In recent years, super-resolution reconstruction technology based on deep learning has made rapid progress. Deep neural networks have demonstrated powerful capabilities in the field of image super-resolution reconstruction, and can produce fast, high-quality high-resolution image results. These include extracting features, upsampling and reconstruction through convolution; generating more realistic high-resolution images through the generative network of GAN, and using adversarial networks to improve the quality of reconstructed images; and using recurrent neural networks (RNNs) to process the time series information of different slices or layers of 3D images to help improve the longitudinal resolution. However, these super-resolution reconstruction algorithms have some limitations: (1) They are fully supervised models and require a large number of high-quality low-resolution-high-resolution image pairs for training. Insufficient data will affect the reconstruction effect. In addition, high-resolution MRI images are difficult to obtain; (2) Deep learning models sometimes generate inaccurate or fictitious details during super-resolution reconstruction. This may result in artifacts or unrealistic anatomical structures in the reconstructed 3D images. For medical applications, false details may affect clinical judgment and bring potential risks. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a medical image registration method based on self-supervised multimodal fusion.

[0007] The purpose of the present invention can be achieved through the following technical solutions.

[0008] A medical image registration method based on self-supervised multimodal fusion, comprising the following steps:

[0009] Obtain a pair of CT and MRI images of the same patient;

[0010] For the original MRI image, a high-resolution MRI image super-resolution reconstruction model is used, which is self-supervised and trained using slices of the coronal and sagittal planes of the MRI image as supervisory signals.

[0011] Taking CT as the fixed image and MRI as the moving image to be deformed, the original MRI and CT images are used to train the first deformation field generation model to obtain a coarse-grained deformation field. The high-resolution MRI and CT images obtained by the super-resolution reconstruction model are used to train the second deformation field generation model to obtain a fine-grained deformation field. The coarse-grained deformation field and the fine-grained deformation field are superimposed to obtain a hybrid deformation field.

[0012] The patient's CT and MRI images are registered using a hybrid deformation field.

[0013] As an optimal technical solution, the super-resolution reconstruction model includes three modules: a feature encoder, an upsampling network, and a feature decoder.

[0014] The feature encoder adopts a student-teacher network. The input original image is down-sampled in the coronal and sagittal planes and then input into the student network of the feature encoder to obtain a low-resolution feature image of each layer. The low-resolution image features of each group of adjacent frames are input into the upsampling network.

[0015] The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN network module to predict the feature map of the upsampling layer of the inserted frame to obtain the synthesized intermediate layer features.

[0016] The decoder module passes the synthesized intermediate layer features through the convolution layer to obtain the predicted high-resolution synthetic image.

[0017] As an optimal technical solution, in the feature encoder using the student-teacher network, both the student network and the teacher network use RDN networks, and the network structure and initial weights are exactly the same.

[0018] The student network input is the MRI image after frame downsampling in the coronal and sagittal planes, and a low-resolution feature image of each layer is obtained, and the gradient is updated through loss backpropagation.

[0019] The teacher network input is the original MRI image, and high-resolution feature maps of the coronal and sagittal planes are obtained; the teacher network gradient is fixed and updated by the weight of the student network through exponential sliding average.

[0020] As an optimal technical solution, the RDN network module is a network feature extractor, including two convolutional layers for extracting shallow features, with multiple RDBs connected after the convolutional layers; the RDB module adopts a densely connected layer structure, allowing each layer to access the feature information of all previous layers; multiple RDB modules are cascaded and the output of each layer is spliced to obtain features of different levels of refinement. The RDB module passes through two convolutional layers and then superimposes the features obtained with the shallowest convolutional layer to obtain the final encoded features.

[0021] As an optimal technical solution, the super-resolution reconstruction model uses the loss between the high-resolution synthetic images of the coronal and sagittal planes predicted by the feature decoder and the coronal and sagittal planes of the original image, as well as the consistency loss between the high-resolution feature map obtained by the teacher network and the feature map output by the sampling network on the student network for joint optimization.

[0022] As a preferred technical solution, the optimization loss of the super-resolution reconstruction model is:

[0023] ,

[0024] in, The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image is calculated. The mean square error loss function is used to ensure the pixel-by-pixel consistency between the interpolated slices and the true intermediate slices. The formula is as follows:

[0025] ,

[0026] in, It is a set of student feature encoder-upsampling network-feature decoder; The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampling network on the student network is calculated. The mean square error loss is used to maintain the consistency of the features of the two branches. The formula is as follows:

[0027] ,

[0028] in, , is the number of pixels in the sagittal and coronal directions of the original MRI image; For teacher coders, For student coders, is an upsampling network; , Respectively downsampled sagittal and coronal slices; , Respectively sagittal and coronal slices of the original image.

[0029] As a preferred technical solution, the generation process of the hybrid deformation field is specifically as follows:

[0030] The original MRI image and the fixed CT image are simultaneously input into the first registration network Obtain a coarse-grained deformation registration field :

[0031] ,

[0032] The high-resolution MRI image and the fixed CT image obtained by the super-resolution reconstruction model are simultaneously input into the second registration network Obtain fine-grained deformation registration field :

[0033] ,

[0034] in, They are CT fixed images and original MRI moving images; To obtain high-resolution MRI images through upsampling on the super-resolution network;

[0035] The two deformation fields 、 Superposition, and use a 3D convolution layer to get the final mixed deformation field .

[0036] As a preferred technical solution, the registration network and The UNet architecture is used. The input is a spliced moving image and a fixed image. 3D features are extracted through multiple layers of 3D convolutional layers and activation layers. After multiple convolution operations, the full-resolution deformation field is refined. .

[0037] As an optimal technical solution, the loss of the registration field fusion is:

[0038] ,

[0039] in, They are CT fixed images and original MRI moving images; Indicates that the blended deformation field superimposed on the original MRI moving image; For the similarity loss between fixed and moving images, a structural similarity index is used to optimize the similarity; Mixed deformation field Gradient norm, is the loss weight.

[0040] As a preferred technical solution, the specific process of registering the MRI image to the CT image is as follows:

[0041] Taking CT as the fixed image and MRI as the moving image that needs to be deformed, an optimized deformation field is obtained based on the two image inputs. The deformation field contains the position change information of all pixels or voxels in the nonlinear registration, so that the moving image is aligned with the fixed image in detail.

[0042] The deformation field is applied to the mask of the moving image to obtain the aligned moving image mask, and the DICE score is calculated with the fixed image mask to evaluate the registration performance.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1) This invention combines deep learning and other artificial intelligence technologies to construct a new head and neck medical image registration method based on self-supervised multimodal fusion. It uses self-supervised training to obtain high-resolution MRI images, and uses the deformation field estimated by the original low-resolution multimodal flow and the synthesized high-resolution multimodal flow to overcome the shortcomings of non-rigid registration. The coarse-grained registration field obtained by low-resolution MRI and CT is fused with the fine-grained registration field generated by high-resolution MRI and CT obtained by super-resolution reconstruction, which improves the accuracy of cross-modal registration, realizes high-precision registration of head and neck MRI and CT images, and allows accurate calibration of images from different modes, enabling doctors to view bone and soft tissue images at the same time, thereby better assessing the extent of tumor invasion.

[0045] To address the issue of low-resolution MRI, this paper integrates a super-resolution reconstruction network. This network improves the quality of MRI images, allowing for the generation of high-resolution outputs from lower-quality inputs. This improvement enables accurate image registration, resulting in higher-quality images that are better aligned and provide more reliable diagnostic information.

[0046] 3) In the algorithm for synthesizing high-resolution images, this paper uses a student-teacher network and a self-supervised learning strategy to super-reconstruct thick-slice MRI images. During training, paired LR-HR images are synthesized in sagittal and coronal directions for pre-training. During inference, sparse axial slices can be used as triplets to guide network refinement. This strategy enables highly accurate registration of CT and MRI images.

[0047] 4) The proposed super-resolution reconstruction network utilizes self-supervised learning techniques, eliminating the need for labels or high-resolution MRI images. It uses coronal and sagittal slices as supervisory signals to train the super-resolution network. This approach is particularly useful in situations where thin-slice MRI images are difficult to obtain, as it can generate high-resolution thin-slice MRI images using a large number of unlabeled thick-slice MRI images.

[0048] 5) This invention employs a multimodal flow fusion network to fuse coarse-grained and fine-grained registration fields to improve the accuracy of cross-modal registration. This approach addresses the computational challenges associated with traditional non-rigid registration methods, which often require extensive optimization and can be resource-intensive. By streamlining the process, this invention improves the speed and efficiency of image registration, making it more suitable for clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the process of the head and neck medical image registration method based on self-supervised multimodal fusion of the present invention;

[0050] Figure 2This is a network diagram of the first stage of super-resolution reconstruction based on self-supervision of the present invention;

[0051] Figure 3 This is the RDB module framework diagram;

[0052] Figure 4 A schematic diagram of the network process for generating a coarse-grained scale deformation field in the second stage of the present invention;

[0053] Figure 5 A detailed diagram of the network structure for the present invention;

[0054] Figure 6 This is the multimodal registration framework diagram of the present invention. DETAILED DESCRIPTION

[0055] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0056] Example 1

[0057] This paper combines artificial intelligence technologies such as deep learning to construct a new head and neck medical image registration method based on self-supervised multimodal fusion. It uses self-supervised training to obtain high-resolution MRI images and fuses coarse-grained and fine-grained registration fields to improve the accuracy of cross-modal registration. A multimodal flow fusion network is proposed, which consists of two parts: (1) using a fixed CT (Fixed image) and a moving MRI (Moving image) to generate a first deformation field; (2) using a super-resolution reconstruction network to upsample the moving MRI (Moving image) and combine it with the fixed CT (Fixed image) to generate a second deformation field. The deformation fields of these two parts are fused to generate a multimodal flow fusion deformation field.

[0058] A multimodal flow fusion approach is used to solve the computationally time-consuming and complex deformation field optimization problem, enabling rapid generation of deformation fields for cross-modal registration. By analyzing and processing paired CT and MRI images of a patient, a paired deformation field is generated. This deformation field is then applied to the MRI, resulting in an MRI image aligned with the CT. The combination of these two modal data is clinically significant for physicians in diagnosing and assessing tumors.

[0059] The head and neck medical image registration method based on self-supervised multimodal fusion proposed in the present invention uses multiple groups of CT and MRI images from the same patient to train the model to generate a deformation field, thereby achieving accurate registration. The present invention includes two processes: model construction and image reasoning, wherein the model construction is divided into two stages: in the first stage, a self-supervised learning model is constructed based on a student-teacher network to generate thin-layer MRI images for thick-layer MRI images; in the second stage, the deformation field obtained by high-resolution MRI-CT is fused with the deformation field obtained by low-resolution MRI-CT to generate a multi-scale combined deformation field, and finally a high-precision registration result is obtained. Figure 1 The specific implementation process is as follows.

[0060] During the diagnosis and treatment of head and neck tumors, CT and MRI scans of the head and neck are acquired from the same patient at different times. Each patient has a pair of CT and MRI images. Using precise image registration technology, the CT and MRI images of the head and neck are aligned, allowing for observation of tumor invasion of bone and soft tissue from the same perspective.

[0061] In the first stage of self-supervised super-resolution reconstruction of MRI images, the original low-resolution MRI images have three planes: axial plane, coronal plane and sagittal plane. The coronal and sagittal planes of the head and neck MRI images have high resolution, while the axial resolution is low due to the influence of acquisition time. Therefore, the self-supervised model of the present invention uses high-resolution slices of the coronal and sagittal planes as supervisory signals to train the self-supervised super-resolution network.

[0062] The first stage model mainly consists of three modules: feature encoder, upsampling network, and feature decoder. Figure 2The feature encoder has two branches, including a student network and a teacher network. The structures and initial weights of the student and teacher networks are exactly the same. The student network performs gradient updates through loss backpropagation, while the teacher network gradient is fixed and updated by the weights of the student network through exponential moving average (EMA). The input original image is downsampled in the coronal and sagittal planes and input into the student feature encoder of the RDN network to obtain low-resolution feature images of each layer. The low-resolution image features of each group of adjacent frames are input into the upsampling network. The upsampling network predicts the feature map of the upsampling layer of the interpolated frame and then sends it to the feature decoder to obtain the synthesized upsampling map. The synthesized coronal and sagittal maps are compared with the coronal and sagittal maps of the original image to calculate the MSE loss. In another teacher network, the original image is input into the teacher encoder of the RDN network to obtain high-resolution feature maps of the coronal and sagittal planes. The consistency loss is calculated for this feature map and the feature map output by the upsampling network on the student network. The consistency loss is combined with the MSE loss to jointly optimize the super-resolution network.

[0063] The RDN module is a network feature extractor, and its internal structure is shown in Figure 3 It includes two convolutional layers to extract shallow features, followed by three RDB (Residual Dense Block) modules. The RDB module allows each layer to access the feature information of all previous layers through a densely connected layer structure, effectively suggesting rich local features and reducing information loss during training. The three RDB modules are cascaded and the output of each layer is concatenated to obtain features of varying degrees of refinement. These features are then passed through two convolutional layers and finally superimposed with the features obtained from the shallowest convolutional layer to obtain the final encoded features. The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN module to predict the intermediate layer features. The decoder module passes the synthesized intermediate layer features through a convolutional layer to obtain a predicted high-resolution synthetic image.

[0064] Specifically, first extract the original MRI image The slices in the coronal and sagittal planes are sliced and the slices in these two directions are downsampled to obtain a synthetic low-resolution image: 、 . Then, from Extract two adjacent sagittal slices as input data for training, and The corresponding real middle slice is obtained as the true value to construct the training data set ,in express The sagittal slices, yes The Using the same operation, we can and The training data set is obtained by coronal slices After two adjacent slices of the synthesized low-resolution image are input into the student feature encoder-upsampling network-feature decoder, the missing intermediate slice can be generated. To optimize the model, the mean squared error (MSE) loss function is used to ensure the pixel-by-pixel consistency between the interpolated slice and the true intermediate slice, and its formula is as follows:

[0065] ,

[0066] in, , is the number of pixels in the sagittal and coronal directions of the original MRI image; The teacher encoder extracts features from the sagittal and coronal slices of the original image to obtain features of the two slices of the original image. Meanwhile, the features of the student encoder are passed through the upsampling network to obtain features of the two slices at the original resolution. The mean square error (MSE) loss is used to maintain the consistency of the features of the two branches. The consistency loss formula is as follows:

[0067] ,

[0068] in, For teacher coders, For student coders, is an upsampling network. The overall optimization loss of the first stage model is:

[0069] ,

[0070] in, The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image, The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampled network on the student network is used. By combining these two losses, we can simultaneously optimize the feature space and the physical space of the real image, achieving super-resolution reconstruction by combining multi-scale features.

[0071] In the second stage, the original low-resolution MRI image and the original CT are used to train a deformation field generation model to obtain a coarse-grained deformation field. The original low-resolution MRI image is then upsampled using the super-resolution model trained in the first stage to obtain a high-resolution MRI image. This is then used with the original CT to train a second deformation field generation model to obtain a fine-grained deformation field. Although the super-resolution model proposed in this invention can effectively upsample MRI images, most super-resolution methods inevitably produce unrealistic details, leading to certain mismatch problems. Therefore, fusing the true coarse-grained deformation field with the fine-grained deformation field with unrealistic details can fully utilize the information of both moving and fixed images.

[0072] like Figure 4 As shown, the network in the second stage consists of four parts: the super-resolution network obtained in the first stage , registration network , registration network and a registration field fusion module.

[0073] Specific registration network and See the structure Figure 5 , which uses the UNet architecture. Each rectangle represents a 3D feature, which is generated from the previous feature through the 3D convolutional network layer. The input of the registration network is the spliced moving image and the fixed image. The features are extracted through multiple layers of 3D convolutional layers and activation layers. After several convolution operations, the full-resolution deformation field is further refined. .

[0074] The input moving MRI image High-resolution MRI images are obtained by upsampling on the super-resolution network ,During the training process of the second stage network, the weights of the first stage super-resolution model remain unchanged. The moving MRI image and the fixed CT image are input into the registration network at the same time. Get the first deformation registration field :

[0075] ,

[0076] Similarly, the upsampled moving MRI image and fixed CT image are simultaneously input into the registration network Get the second deformation registration field :

[0077] ,

[0078] The two deformation fields 、 Superposition, and use the convolution kernel size of The three-dimensional convolution layer is used to obtain the final mixed deformation field Therefore, the loss of the network at this stage is:

[0079] ,

[0080] in, They are fixed CT images and original moving MRI images, To transform the mixed deformation field Superimposed on the moving MRI image. The first loss is a similarity loss that penalizes the appearance difference between fixed and moving images, and uses the structural similarity index (SIIM) to optimize the similarity. Using mixed deformation field Gradient norm. is the loss weight.

[0081] Finally, the mixed deformation field obtained in the second stage , taking the registration of MRI to CT images of the same patient as an example, Figure 6 As shown in Figure 2, the registration process is as follows:

[0082] (1) CT is a fixed image, and MRI is an image that needs to be deformed. The two images are input into the model of the present invention to obtain an optimized deformation field, so that the moving image is aligned with the fixed image in detail.

[0083] (2) The pairing process yields a deformation field, which describes how each pixel or voxel in the image transforms from the source image (Moving Image) to the target image (Fixed Image). The deformation field contains information about the positional changes of all pixels or voxels in the nonlinear registration and is the core of achieving fine alignment.

[0084] (3) Apply the deformation field to the moving image mask to obtain the aligned moving image label, and calculate the dice score with the fixed image label to evaluate the registration performance.

[0085] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0086] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A medical image registration method based on self-supervised multimodal fusion, characterized in that the steps include: Obtain a pair of CT and MRI images of the same patient; For the original MRI image, a high-resolution MRI image super-resolution reconstruction model is used, which is self-supervised and trained using slices of the coronal and sagittal planes of the MRI image as supervisory signals. The super-resolution reconstruction model includes three modules: feature encoder, upsampling network, and feature decoder; The feature encoder uses a student-teacher network. The input original image is downsampled in the coronal and sagittal planes and then input into the student network of the feature encoder to obtain a low-resolution feature image of each layer. The low-resolution image features of each group of adjacent frames are input into the upsampling network. The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN network module to predict the feature map of the upsampling layer of the inserted frame, thereby obtaining the synthesized intermediate layer features. The decoder module passes the synthesized intermediate layer features through the convolution layer to obtain the predicted high-resolution synthetic image; In the feature encoder using the student-teacher network, both the student network and the teacher network use RDN networks, and the network structure and initial weights are exactly the same; The original MRI image is downsampled in the coronal and sagittal planes and input into the student network feature encoder to obtain low-resolution feature images at each layer. The low-resolution image features of each group of adjacent frames are input into the upsampling network. The upsampling network predicts the feature map of the upsampling layer of the interpolated frame, which is then sent to the feature decoder to obtain the synthesized upsampling map. The synthesized coronal and sagittal maps are compared with the coronal and sagittal maps of the original image to calculate the MSE loss. The student network performs gradient update based on the MSE loss. The original MRI image is input into the teacher network feature encoder to obtain high-resolution feature maps in the coronal and sagittal planes. The feature maps are compared with the feature maps output by the student network to calculate the consistency loss. The super-resolution network is optimized jointly by the consistency loss and the MSE loss. The teacher network gradient is fixed and updated by the weights of the student network through the exponential sliding average. The super-resolution reconstruction model uses the loss between the coronal and sagittal high-resolution synthetic images predicted by the feature decoder and the coronal and sagittal images of the original image, as well as the consistency loss between the high-resolution feature map obtained by the teacher network and the feature map output by the sampling network on the student network for joint optimization; The optimization loss of the super-resolution reconstruction model is: Among them, l MSE The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image is calculated. The mean square error loss function is used to ensure the pixel-by-pixel consistency between the interpolated slices and the true intermediate slices. The formula is as follows: in, It is a set of student feature encoder-upsampling network-feature decoder; l consistency The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampling network on the student network is calculated. The mean square error loss is used to maintain the consistency of the features of the two branches. The formula is as follows: in, For teacher coders, For student coders, is an upsampling network; are the sagittal and coronal slices of the i-th downsample, respectively; are the sagittal and coronal slices of the i-th original image respectively; These are the sagittal and coronal slices of the 2ith original image, respectively; are the sagittal and coronal slices of the i+1th downsample, respectively; X, Y are the number of pixels in the sagittal and coronal directions of the original MRI image; Taking CT as the fixed image and MRI as the moving image to be deformed, the original MRI and CT images are used to train the first deformation field generation model to obtain a coarse-grained deformation field. The high-resolution MRI and CT images obtained by the super-resolution reconstruction model are used to train the second deformation field generation model to obtain a fine-grained deformation field. The coarse-grained deformation field and the fine-grained deformation field are superimposed to obtain a hybrid deformation field. The patient's CT and MRI images are registered using a hybrid deformation field.

2. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The RDN network module is a network feature extractor, which includes two convolutional layers for extracting shallow features, and multiple RDBs are connected after the convolutional layers. The RDB module adopts a densely connected layer structure, allowing each layer to access the feature information of all previous layers. Multiple RDB modules are cascaded and the output of each layer is spliced to obtain features of different levels of refinement. The RDB module passes through two convolutional layers and then superimposes the features obtained with the shallowest convolutional layer to obtain the final encoded features.

3. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The generation process of the hybrid deformation field is specifically as follows: The original MRI image and the fixed CT image are simultaneously input into the first registration network Get the coarse-grained deformation registration field φ1: Among them, f and m are the fixed CT image and the original moving MRI image respectively; The high-resolution MRI image and the fixed CT image obtained by the super-resolution reconstruction model are simultaneously input into the second registration network Get the fine-grained deformation registration field φ2: The two deformation fields φ1 and φ2 are superimposed, and a three-dimensional convolution layer is used to obtain the final mixed deformation field φ3.

4. The medical image registration method based on self-supervised multimodal fusion according to claim 3, characterized in that: The registration network and The UNet architecture is adopted, with the input being the spliced moving image and the fixed image. 3D features are extracted through multiple layers of 3D convolutional layers and activation layers. After multiple convolution operations, the full-resolution deformation field φ is refined.

5. The medical image registration method based on self-supervised multimodal fusion according to claim 3, characterized in that: The loss of the registration field fusion is: Among them, f and m are CT fixed image and original MRI moving image respectively; It means superimposing the mixed deformation field φ3 onto the original MRI moving image; For the similarity loss between fixed and moving images, a structural similarity index is used to optimize the similarity; is the L2 norm of the gradient of the mixed deformation field φ3, and λ is the loss weight.

6. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The specific process of registering the MRI image to the CT image is as follows: Taking CT as the fixed image and MRI as the moving image to be deformed, an optimized deformation field is obtained based on the two image inputs. The deformation field contains the position change information of all pixels or voxels in the nonlinear registration, so that the moving image is aligned with the fixed image in detail. The deformation field is applied to the mask of the moving image to obtain the aligned moving image mask, and the DICE score is calculated with the fixed image mask to evaluate the registration performance.

Images