Medical image registration method based on self-supervised multi-modal fusion

Abstract

The invention relates to a medical image registration method based on self-supervised multi-modal fusion. The method comprises the following steps: acquiring a pair of CT and MRI images of the same patient; adopting an MRI image super-resolution reconstruction model which takes slices of a coronal plane and a sagittal plane of an MRI image as supervision signals to carry out self-supervision training, and obtaining a high-resolution MRI image; a CT is used as a fixed image, an MRI is used as a moving image needing to be deformed, and a deformation field generation model is trained by using a high-resolution MRI image and a CT image obtained by training an original MRI image and a CT image and reconstructing a model to obtain a coarse-grained-scale deformation field and a fine-grained-scale deformation field; and the two deformation fields are superposed to obtain a mixed deformation field to carry out registration on the CT and MRI images of the patient. Compared with the prior art, the method has the advantages that calculation challenges related to a traditional non-rigid registration method are solved, the image registration speed and efficiency are improved, and the method is more suitable for clinical application.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image registration, and in particular to a medical image registration method based on self-supervised multimodal fusion. Background Art

[0002] In the diagnosis and treatment of head and neck tumors, it is crucial to accurately obtain the spatial distribution and tissue type of the tumor. Since the tumor may invade both soft tissue and bone tissue at the same time, in complex cases, a single modality may not be able to fully show the full picture of the lesion, so a variety of imaging techniques are needed for comprehensive evaluation. Computed tomography (CT) images have better imaging effects on bones and high-density tissues, while magnetic resonance imaging (MRI) performs better on soft tissues (such as muscles, brain, organs, etc.). However, for the image data of these two modalities, doctors need to check the invasion of the tumor on both images at the same time when making a diagnosis. Unaligned dual-modality images are often unable to be accurately aligned due to factors such as changes in patient position and differences in equipment imaging angles, resulting in doctors being unable to directly compare the features of the same anatomical structure. Because obtaining structurally aligned dual-modality images is particularly important for accurately locating lesions and surrounding key structures, it optimizes the process of diagnosis, evaluation and treatment of lesions.

[0003] The process of aligning two or more images to the same coordinate system is called image registration, which enables the corresponding structures in the images to be accurately matched. The main goal of this invention is to achieve bidirectional registration between CT images and MRI images, that is, to register CT images to MRI images, and vice versa. This bidirectional registration method can provide more comprehensive diagnostic information, helping doctors to observe the imaging data of bone tissue and soft tissue at the same viewing angle, so as to accurately assess the spread and invasion of tumors.

[0004] Traditional image registration often uses non-rigid deformation image registration algorithms to obtain deformation fields, which involves complex optimization processes and large-scale deformation field solutions, consumes large computing resources, and runs slowly. When processing large-size, high-resolution images, the calculation time is long, which is especially inefficient in real-time applications or large-scale data processing scenarios. In addition, the performance of non-rigid deformation-based registration algorithms depends largely on parameter settings, such as the smoothness of the deformation field, regularization weights, etc. If the parameters are set improperly, it may lead to overfitting or underfitting, affecting the quality of the registration results.

[0005] MRI scans require a long time for data collection, usually tens of minutes or even longer. If you want to improve the resolution, you need to increase the scanning time to improve the signal-to-noise ratio. However, too long a scanning time is not only uncomfortable for the patient, but also easily causes artifacts due to unconscious movements (such as breathing, small movements), which in turn affects the image quality. In recent years, super-resolution reconstruction technology based on deep learning has made rapid progress. Deep neural networks have demonstrated powerful capabilities in the field of image super-resolution reconstruction and can produce fast, high-quality high-resolution image results. This includes extracting features, upsampling and reconstruction through convolution; generating more realistic high-resolution images through the generative network of GAN, and using adversarial networks to improve the quality of reconstructed images; using recurrent neural networks (RNNs) to process the time series information of different slices or layers of 3D images to help improve the longitudinal resolution. However, these super-resolution reconstruction algorithms have some limitations: (1) They are fully supervised models, which require a large number of high-quality low-resolution-high-resolution image pairs for training. Insufficient data will affect the reconstruction effect. In addition, high-resolution MRI images are difficult to obtain; (2) Deep learning models sometimes generate inaccurate or fictitious details during super-resolution reconstruction. This may result in artifacts or unrealistic anatomical structures in the reconstructed 3D images. For medical applications, false details may affect clinical judgment and bring potential risks. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a medical image registration method based on self-supervised multimodal fusion.

[0007] The purpose of the present invention can be achieved through the following technical solutions.

[0008] A medical image registration method based on self-supervised multimodal fusion, comprising the following steps: Obtain a pair of CT and MRI images of the same patient; For the original MRI image, a high-resolution MRI image is obtained by using the slices of the coronal and sagittal planes of the MRI image as supervisory signals for self-supervisory training of the MRI image super-resolution reconstruction model; Taking CT as a fixed image and MRI as a moving image that needs to be deformed, the original MRI image and CT image are used to train the first deformation field generation model to obtain a coarse-grained deformation field; the high-resolution MRI image and CT image obtained by the super-resolution reconstruction model are used to train the second deformation field generation model to obtain a fine-grained deformation field; the coarse-grained deformation field and the fine-grained deformation field are superimposed to obtain a mixed deformation field; The patient's CT and MRI images are registered using a hybrid deformation field.

[0009] As a preferred technical solution, the super-resolution reconstruction model includes three modules: a feature encoder, an upsampling network, and a feature decoder.

[0010] The feature encoder adopts a student-teacher network. The input original image is down-sampled in the coronal and sagittal planes and then input into the student network of the feature encoder to obtain a low-resolution feature image of each layer, and the low-resolution image features of each group of adjacent frames are input into the upsampling network.

[0011] The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN network module to predict the feature map of the upsampling layer of the inserted frame to obtain the synthesized intermediate layer features.

[0012] The decoder module passes the synthesized intermediate layer features through the convolution layer to obtain the predicted high-resolution synthetic image.

[0013] As a preferred technical solution, in the feature encoder using the student-teacher network, both the student network and the teacher network use RDN networks, and the network structure and initial weights are exactly the same.

[0014] The student network input is the MRI image in the coronal and sagittal planes after frame downsampling, and a low-resolution feature image of each layer is obtained, and the gradient is updated through loss feedback.

[0015] The teacher network input is the original MRI image, and high-resolution feature maps of the coronal and sagittal planes are obtained; the teacher network gradient is fixed and updated by the weight of the student network through exponential sliding average.

[0016] As a preferred technical solution, the RDN network module is a network feature extractor, including two convolutional layers for extracting shallow features, and multiple RDBs are connected after the convolutional layers; the RDB module adopts a densely connected layer structure, allowing each layer to access the feature information of all previous layers; multiple RDB modules are cascaded and the output of each layer is spliced ​​to obtain features of different degrees of refinement. The RDB module passes through two convolutional layers and then superimposes the features obtained with the shallowest convolutional layer to obtain the final encoding features.

[0017] As a preferred technical solution, the super-resolution reconstruction model uses the loss between the high-resolution synthetic images of the coronal and sagittal planes predicted by the feature decoder and the coronal and sagittal planes of the original image, as well as the consistency loss between the high-resolution feature map obtained by the teacher network and the feature map output by the sampling network on the student network for joint optimization.

[0018] As a preferred technical solution, the optimization loss of the super-resolution reconstruction model is: , in, The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image is used. The mean square error loss function is used to ensure the pixel-by-pixel consistency between the interpolated slices and the true intermediate slices. The formula is as follows: , in, It is a collection of student feature encoder-upsampling network-feature decoder; The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampling network on the student network is calculated. The mean square error loss is used to maintain the consistency of the features of the two branches. The formula is as follows: , in, , is the number of pixels in the sagittal and coronal directions of the original MRI image; For teacher coders, For student coders, is an upsampling network; , Respectively downsampled sagittal and coronal slices; , Respectively Sagittal and coronal slices of the original image.

[0019] As a preferred technical solution, the generation process of the hybrid deformation field is specifically as follows:

[0020] The original MRI image and the fixed CT image are simultaneously input into the first registration network Get the coarse-grained deformation registration field : , The high-resolution MRI image and the fixed CT image obtained by the super-resolution reconstruction model are simultaneously input into the second registration network Get a fine-grained deformation registration field : , in, They are CT fixed images and original MRI moving images; To obtain high-resolution MRI images through upsampling on the super-resolution network; The two deformation fields , Superposition, and use a 3D convolution layer to get the final mixed deformation field .

[0021] As a preferred technical solution, the registration network and The UNet architecture is used. The input is a spliced ​​moving image and a fixed image. The 3D features are extracted through multiple layers of 3D convolutional layers and activation layers. After multiple convolution operations, the full-resolution deformation field is refined. .

[0022] As a preferred technical solution, the loss of the registration field fusion is: , in, They are CT fixed images and original MRI moving images; Indicates that the blended deformation field Superimposed on the original MRI moving image; For the similarity loss between fixed and moving images, a structural similarity index is used to optimize the similarity; Mixed deformation field Gradient norm, is the loss weight.

[0023] As a preferred technical solution, the specific process of registering the MRI image to the CT image is as follows:

[0024] Taking CT as a fixed image and MRI as a moving image that needs to be deformed, an optimized deformation field is obtained based on the two image inputs. The deformation field contains the position change information of all pixels or voxels in the nonlinear registration, so that the moving image is aligned with the fixed image in detail.

[0025] The deformation field is applied to the mask of the moving image to obtain the aligned moving image mask, and the DICE score is calculated with the fixed image mask to evaluate the registration performance.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1) The present invention combines artificial intelligence technologies such as deep learning to construct a new head and neck medical image registration method based on self-supervised multimodal fusion. It uses self-supervised training to obtain high-resolution MRI images, and uses the deformation field estimated by the original low-resolution multimodal flow and the synthesized high-resolution multimodal flow to overcome the shortcomings of non-rigid registration. The coarse-grained registration field obtained by low-resolution MRI and CT is fused with the fine-grained registration field generated by high-resolution MRI and CT obtained by super-resolution reconstruction, which improves the accuracy of cross-modal registration, realizes high-precision registration of head and neck MRI and CT images, and allows accurate calibration of images from different modes, allowing doctors to view bone and soft tissue images at the same time, thereby better assessing the extent of tumor invasion.

[0028] 2 In order to solve the problem of low-resolution magnetic resonance imaging, the present invention integrates a super-resolution reconstruction network. The network improves the quality of MRI images, allowing high-resolution output to be generated from lower-quality inputs. This improvement can obtain higher-quality images for accurate image registration, which can be better aligned and provide more reliable diagnostic information.

[0029] 3) In the algorithm for synthesizing high-resolution images, the present invention uses a student-teacher network and a self-supervised learning strategy to perform super-resolution reconstruction on thick MRI images. In the training, paired LR-HR images are synthesized based on sagittal and coronal directions for pre-training. During the inference process, sparse axial slices can be used as a useful guide for network refinement in a triplets manner. Based on this strategy, high-precision registered CT and MRI images can be obtained.

[0030] 4) The super-resolution reconstruction network proposed in the present invention uses self-supervised learning technology, without any labels and high-resolution MRI images, and uses slices in both the coronal and sagittal planes as supervisory signals to train the super-resolution network. This method is very suitable for situations where thin-layer MRI images are difficult to obtain, and uses a large number of unlabeled thick-layer MRIs to generate high-resolution thin-layer MRI images.

[0031] 5) The present invention uses a multimodal flow fusion network to fuse coarse-grained and fine-grained registration fields to improve the accuracy of cross-modal registration. This approach addresses the computational challenges associated with traditional non-rigid registration methods, which often require extensive optimization and can be resource-intensive. By simplifying the process, the present invention improves the speed and efficiency of image registration, making it more suitable for clinical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of the head and neck medical image registration method based on self-supervised multimodal fusion of the present invention; Figure 2 A schematic diagram of a network for super-resolution reconstruction based on self-supervision in the first stage of the present invention; Figure 3 This is the RDB module framework diagram; Figure 4 A schematic diagram of the network process for generating a coarse-grained scale deformation field in the second stage of the present invention; Figure 5 A detailed diagram of the network structure for the present invention; Figure 6 This is a diagram of the multimodal registration framework of the present invention. DETAILED DESCRIPTION

[0033] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0034] Example 1 This paper combines artificial intelligence technologies such as deep learning to construct a new head and neck medical image registration method based on self-supervised multimodal fusion. It uses self-supervised training to obtain high-resolution MRI images, and fuses coarse-grained and fine-grained registration fields to improve the accuracy of cross-modal registration. A multimodal flow fusion network is proposed, which includes two parts: (1) using fixed CT (Fixed image) and moving MRI (Moving image) to generate the first deformation field; (2) using a super-resolution reconstruction network to upsample the moving MRI (Moving image) and combine it with the fixed CT (Fixed image) to generate the second deformation field. The deformations of these two parts are fused to generate a multimodal flow fusion deformation field.

[0035] A method based on multimodal flow fusion is used to solve the time-consuming and complex deformation field optimization problem to quickly generate deformation fields for cross-modal registration. By analyzing and processing the paired CT and MRI images of the patient, a paired deformation field can be generated, and this deformation field is applied to the MRI to obtain an MRI image aligned with the CT. The combination of the two modal data has important clinical significance for doctors to diagnose and evaluate tumors.

[0036] The head and neck medical image registration method based on self-supervised multimodal fusion proposed in the present invention uses multiple groups of CT and MRI images from the same patient to train the model to generate a deformation field, thereby achieving accurate registration. The present invention includes two processes: model construction and image reasoning, wherein the model construction is divided into two stages: in the first stage, a self-supervised learning model is constructed based on a student-teacher network to generate thin-layer MRI images for thick-layer MRI images; in the second stage, the deformation field obtained by high-resolution MRI-CT is fused with the deformation field obtained by low-resolution MRI-CT to generate a multi-scale combined deformation field, and finally a high-precision registration result is obtained. Figure 1 As shown, the specific implementation process is as follows.

[0037] During the diagnosis and treatment of head and neck tumors, CT and MRI scans of the head and neck taken at different times are obtained for the same patient. Each patient has a pair of CT and MRI images. Through precise image registration technology, the alignment of head and neck CT and MRI images is achieved, so that the invasion of tumors on bone tissue and soft tissue can be observed from the same perspective.

[0038] In the first stage of super-resolution reconstruction of MRI images based on self-supervision, the original low-resolution MRI images have three planes: axial plane, coronal plane and sagittal plane. The coronal and sagittal planes of the head and neck MRI images have high resolution, but the axial resolution is low due to the influence of acquisition time. Therefore, the self-supervised model of the present invention uses high-resolution slices of the coronal and sagittal planes as supervisory signals to train the self-supervised super-resolution network.

[0039] The model in the first stage mainly consists of three modules: feature encoder, upsampling network, and feature decoder. Figure 2 The feature encoder has two branches, including a student network and a teacher network. The structure and initial weights of the student and teacher networks are exactly the same. The student network updates the gradient through loss feedback, while the teacher network gradient is fixed and updated by the weight of the student network through exponential moving average (EMA). The input original image is downsampled in the coronal and sagittal planes and input into the student feature encoder of the RDN network to obtain low-resolution feature images of each layer. The low-resolution image features of each group of adjacent frames are input into the upsampling network. The feature map of the upsampling layer of the interpolated frame is predicted through the upsampling network, and then sent to the feature decoder to obtain the synthesized upsampling map. The synthesized coronal and sagittal maps are compared with the coronal and sagittal maps of the original image to calculate the MSE loss. In another teacher network, the original image is input into the teacher encoder of the RDN network to obtain high-resolution feature maps of the coronal and sagittal planes. The consistency loss is calculated for this feature map and the feature map output by the upsampling network on the student network, which is combined with the MSE loss to optimize the super-resolution network.

[0040] The RDN module is a network feature extractor, and its internal structure is shown in Figure 3 , which includes two convolutional layers to extract shallow features, followed by three RDB (Residual Dense Block) modules. The RDB module allows each layer to access the feature information of all previous layers through a densely connected layer structure, thereby effectively prompting rich local features and reducing information loss during training. The three RDB modules are cascaded and concatenated with the output of each layer to obtain features of different levels of refinement, which are then passed through two convolutional layers and finally superimposed with the features obtained from the shallowest convolutional layer to obtain the final encoded features. The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN module to predict the features of the intermediate layer. The decoder module passes the synthesized intermediate layer features through a convolutional layer to obtain a predicted high-resolution synthetic image.

[0041] Specifically, first extract the original MRI image The slices in the coronal and sagittal planes are sliced ​​and the slices in these two directions are downsampled to obtain a synthetic low-resolution image: , . Then, from Extract two adjacent sagittal slices as input data for training, and The corresponding real middle slice is obtained as the true value to construct the training data set ,in express The Sagittal slices, yes The Using the same procedure, we can and The coronal slices of the training data set are After two adjacent slices of the synthesized low-resolution image are input into the student feature encoder-upsampling network-feature decoder, the missing intermediate slice can be generated. In order to optimize the model, the mean square error (MSE) loss function is used to ensure the pixel-by-pixel consistency between the interpolated slice and the true intermediate slice, and its formula is as follows: , in, , is the number of pixels in the sagittal and coronal directions of the original MRI image; It is a set of student feature encoder-upsample network-feature decoder. The teacher encoder extracts features from the sagittal slice and coronal slice of the original image to obtain the features of the two slices of the original image. At the same time, the features of the student encoder are passed through the upsample network to obtain the two slice features of the original resolution. The mean square error (MSE) loss is used to maintain the consistency of the two branch features. Therefore, the formula for calculating the consistency loss is as follows: , in, For teacher coders, For student coders, is an upsampling network. The overall optimization loss of the first stage model is: , in, The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image, The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampling network on the student network is obtained. By combining these two losses, the feature space and the physical space of the real image can be optimized at the same time, achieving super-resolution reconstruction combining multi-scale features.

[0042] In the second stage, the deformation field generation model is trained using the original low-resolution MRI image and the original CT to obtain a coarse-grained deformation field, and then the original low-resolution MRI image is upsampled using the super-resolution model trained in the first stage to obtain a high-resolution MRI image, and then the second deformation field generation model is trained with the original CT to obtain a fine-grained deformation field. Although the super-resolution model proposed in the present invention can upsample MRI images well, most super-resolution methods will inevitably produce unreal details, resulting in some mismatch problems. Therefore, the real coarse-grained deformation field is fused with the fine-grained deformation field with unreal details, which can make full use of the information of the moving image and the fixed image.

[0043] like Figure 4 As shown, the network in the second stage consists of four parts: the super-resolution network obtained in the first stage , Registration Network , Registration Network and a registration field fusion module.

[0044] Specific registration network and Structure Figure 5 , which uses the UNet architecture. Each rectangle represents a 3D feature, which is generated from the previous feature through a 3D convolutional network layer. The input of the registration network is the spliced ​​moving image and the fixed image. The features are extracted through multiple layers of 3D convolutional layers and activation layers. After several convolution operations, the full-resolution deformation field is further refined. .

[0045] The input mobile MRI image High-resolution MRI images are obtained by upsampling on the super-resolution network , during the training of the second stage network, the weights of the first stage super-resolution model remain unchanged. The mobile MRI image and the fixed CT image are simultaneously input into the registration network Get the first deformation registration field : , Similarly, the upsampled mobile MRI image and fixed CT image are simultaneously input into the registration network Get the second deformation registration field : , The two deformation fields , Superposition, and use the convolution kernel size of The three-dimensional convolution layer is used to obtain the final mixed deformation field . Therefore, the loss of the network optimized at this stage is: , in, They are the fixed CT image and the original moving MRI image, To transform the mixed deformation field Superimposed onto the moving MRI image. The first loss is a similarity loss that penalizes the difference in appearance between fixed and moving images, and uses the structural similarity index (SIIM) to optimize similarity. The second loss Hybrid deformation field Gradient Norm. is the loss weight.

[0046] Finally, the hybrid deformation field obtained in the second stage , taking the MRI registration to CT image of the same patient as an example, Figure 6 As shown in the figure, the registration process is as follows:

[0047] (1) CT is a fixed image, and MRI is an image that needs to be deformed. The two images are input into the model of the present invention to obtain an optimized deformation field, so that the moving image is aligned with the fixed image in detail.

[0048] (2) Through the pairing process, a deformation field can be obtained, which describes how each pixel or voxel in the image is transformed from the source image (Moving Image) to the target image (Fixed Image). The deformation field contains the position change information of all pixels or voxels in nonlinear registration and is the core of achieving fine alignment.

[0049] (3) The deformation field is applied to the moving image mask to obtain the aligned moving image label, and the dice score is calculated with the fixed image label to evaluate the registration performance.

[0050] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0051] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A medical image registration method based on self-supervised multimodal fusion, characterized in that the steps include: Obtain a pair of CT and MRI images of the same patient; For the original MRI image, a high-resolution MRI image is obtained by using the slices of the coronal and sagittal planes of the MRI image as supervisory signals for self-supervisory training of the MRI image super-resolution reconstruction model; Taking CT as a fixed image and MRI as a moving image that needs to be deformed, the original MRI image and CT image are used to train the first deformation field generation model to obtain a coarse-grained deformation field; the high-resolution MRI image and CT image obtained by the super-resolution reconstruction model are used to train the second deformation field generation model to obtain a fine-grained deformation field; The coarse-grained deformation field and the fine-grained deformation field are superimposed to obtain a mixed deformation field. The patient's CT and MRI images are registered using a hybrid deformation field.

2. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The super-resolution reconstruction model includes three modules: feature encoder, upsampling network, and feature decoder; The feature encoder adopts a student-teacher network. The input original image is down-sampled in the coronal plane and the sagittal plane, and then input into the student network of the feature encoder to obtain a low-resolution feature image of each layer, and the low-resolution image features of each group of adjacent frames are input into the upsampling network; The upsampling network concatenates the feature maps of two adjacent layers and inputs them into an RDN network module to predict the feature map of the upsampling layer of the inserted frame to obtain the synthesized intermediate layer features; The decoder module passes the synthesized intermediate layer features through the convolution layer to obtain the predicted high-resolution synthetic image.

3. The medical image registration method based on self-supervised multimodal fusion according to claim 2, characterized in that: In the feature encoder using the student-teacher network, both the student network and the teacher network use the RDN network, and the network structure and initial weights are exactly the same; The student network input is the MRI image after frame downsampling in the coronal and sagittal planes, and a low-resolution feature image of each layer is obtained, and the gradient is updated through loss feedback; The teacher network input is the original MRI image, and high-resolution feature maps of the coronal and sagittal planes are obtained; the teacher network gradient is fixed and updated by the weight of the student network through exponential sliding average.

4. The medical image registration method based on self-supervised multimodal fusion according to claim 3, characterized in that: The RDN network module is a network feature extractor, including two convolutional layers for extracting shallow features, and multiple RDBs are connected after the convolutional layers; the RDB module adopts a densely connected layer structure, allowing each layer to access the feature information of all previous layers; multiple RDB modules are cascaded and the output of each layer is spliced ​​to obtain features of different degrees of refinement. The RDB module passes through two convolutional layers and then superimposes the features obtained with the shallowest convolutional layer to obtain the final encoding features.

5. The medical image registration method based on self-supervised multimodal fusion according to claim 3, characterized in that: The super-resolution reconstruction model uses the loss between the coronal and sagittal high-resolution synthetic images predicted by the feature decoder and the coronal and sagittal images of the original image, as well as the consistency loss between the high-resolution feature map obtained by the teacher network and the feature map output by the sampling network on the student network for joint optimization.

6. The medical image registration method based on self-supervised multimodal fusion according to claim 5, characterized in that: The optimization loss of the super-resolution reconstruction model is: , in, The loss between the coronal and sagittal images synthesized by the student network and the coronal and sagittal images of the original image is used. The mean square error loss function is used to ensure the pixel-by-pixel consistency between the interpolated slices and the true intermediate slices. The formula is as follows: , in, It is a collection of student feature encoder-upsampling network-feature decoder; The consistency loss between the feature map obtained by the teacher network and the feature map output by the sampling network on the student network is calculated. The mean square error loss is used to maintain the consistency of the features of the two branches. The formula is as follows: , in, , is the number of pixels in the sagittal and coronal directions of the original MRI image; For teacher coders, For student coders, is an upsampling network; , Respectively downsampled sagittal and coronal slices; , Respectively Sagittal and coronal slices of the original image.

7. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The generation process of the mixed deformation field is specifically as follows: The original MRI image and the fixed CT image are simultaneously input into the first registration network Get the coarse-grained deformation registration field : , The high-resolution MRI image and the fixed CT image obtained by the super-resolution reconstruction model are simultaneously input into the second registration network Get a fine-grained deformation registration field : , in, They are CT fixed images and original MRI moving images; To obtain high-resolution MRI images through upsampling on the super-resolution network; The two deformation fields , Superposition, and use a 3D convolution layer to get the final mixed deformation field .

8. The medical image registration method based on self-supervised multimodal fusion according to claim 7, characterized in that: The registration network and The UNet architecture is used. The input is a spliced ​​moving image and a fixed image. The 3D features are extracted through multiple layers of 3D convolutional layers and activation layers. After multiple convolution operations, the full-resolution deformation field is refined. .

9. The medical image registration method based on self-supervised multimodal fusion according to claim 7, characterized in that: The loss of the registration field fusion is: , in, They are CT fixed images and original MRI moving images; Indicates that the blended deformation field Superimposed on the original MRI moving image; For the similarity loss between fixed and moving images, a structural similarity index is used to optimize the similarity; Mixed deformation field Gradient norm, is the loss weight.

10. The medical image registration method based on self-supervised multimodal fusion according to claim 1, characterized in that: The specific process of registering the MRI image to the CT image is as follows: Taking CT as a fixed image and MRI as a moving image that needs to be deformed, an optimized deformation field is obtained based on the two image inputs. The deformation field contains the position change information of all pixels or voxels in the nonlinear registration, so that the moving image is aligned with the fixed image in detail. The deformation field is applied to the mask of the moving image to obtain the aligned moving image mask, and the DICE score is calculated with the fixed image mask to evaluate the registration performance.

Images