A multi-modal medical image registration method and system based on a segmentation large model

Abstract

The present application belongs to the technical field of image registration, in order to solve the problems of poor robustness to non-rigid deformation and inaccurate registration in existing medical image registration, a multi-modal medical image registration method and system based on a segmentation large model are proposed, taking the preoperative prostate three-dimensional image as the moving image and the intraoperative prostate three-dimensional image as the target image; the segmentation large model is used to extract the moving mask and the target mask from the moving image and the target image respectively; the moving image, the target image, the moving mask and the target mask are fused to obtain a fusion information tensor; based on the fusion information tensor, a deformable registration network is used to predict a three-dimensional space deformation field, and the moving image is spatially transformed so that the moving image is aligned with the target image after deformation, improving the robustness to non-rigid deformation, achieving high-precision registration and significantly reducing the need for manual annotation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field related to image registration, and particularly relates to a multi-modal medical image registration method and system based on a segmentation large model. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In the preoperative planning and intraoperative navigation process of prostate surgery, accurately registering different modalities of medical images such as magnetic resonance imaging (MRI) and transrectal ultrasound imaging (TRUS) is a key task. Multi-modal medical image registration aims to spatially align images from different imaging devices so that doctors can integrate the information provided by different modalities of images to more accurately locate lesions, plan surgical paths, and assess surgical risks.

[0004] However, this process faces many challenges. First, MRI and TRUS images have significant differences in imaging principles, resolution, and contrast. MRI can provide high-resolution soft tissue contrast, which helps to clearly show the anatomical structure of the prostate and the location of the lesion; while TRUS images have the advantage of real-time imaging, but have lower resolution and are greatly affected by patient physiological conditions and operator skill levels. Second, the prostate may undergo non-rigid deformation before and during surgery, which may be caused by factors such as bladder filling level, intestinal gas, patient position change, and surgical operation. In addition, there may be noise and artifacts in TRUS images, further increasing the difficulty of registration.

[0005] Traditional multi-modal medical image registration methods mainly rely on feature-based registration algorithms or intensity-based registration algorithms. Feature-based registration algorithms extract feature points, edges or contours, etc. in the image, and then find the correspondence between these features to achieve registration. However, these methods require high accuracy of feature extraction and uniformity of feature point distribution, and are prone to matching errors when dealing with non-rigid deformation and noise. Intensity-based registration algorithms minimize the intensity difference between images to find the best registration parameters, but this type of method usually has large computational load and is prone to local optimal solution, especially in the case of large deformation and noise between images.

[0006] In recent years, with the development of deep learning technology, some deep learning-based registration methods have been proposed. These methods can better handle the nonlinear deformation and noise between images by learning the feature representation of the images, and have higher registration accuracy and robustness than traditional registration methods. However, most existing deep learning registration methods require a large amount of labeled data for training, which is often difficult to obtain in the medical image field because the labeling of medical images requires professional medical knowledge and a large amount of time cost. In addition, these methods still have certain limitations in dealing with specific non-rigid deformation and multi-modal image registration in prostate surgery, such as insufficient robustness to image noise and the need to improve registration accuracy.

[0007] Therefore, how to develop a method that can efficiently and accurately handle multi-modal medical image registration in prostate surgery while reducing the dependence on labeled data and improving the robustness to noise and non-rigid deformation is a technical problem that needs to be solved at present. SUMMARY

[0008] To overcome the shortcomings of the prior art, the present application provides a multi-modal medical image registration method and system based on a segmentation large model, which improves the robustness to non-rigid deformation, achieves high-precision registration, and significantly reduces the need for manual labeling.

[0009] To achieve the above purpose, the present application adopts the following technical solutions:

[0010] In a first aspect, the present application provides a multi-modal medical image registration method based on a segmentation large model, comprising:

[0011] Obtaining a preoperative prostate three-dimensional image and an intraoperative prostate three-dimensional image as a pair of images to be registered;

[0012] Inputting the pair of images to be registered into a trained registration model to obtain a registration result of the pair of images to be registered;

[0013] Wherein, the processing process of the registration model for the pair of images to be registered is:

[0014] Taking the preoperative prostate three-dimensional image as a moving image and the intraoperative prostate three-dimensional image as a target image;

[0015] Using a segmentation large model to extract a moving mask and a target mask from the moving image and the target image, respectively;

[0016] Fusing the moving image, the target image, the moving mask and the target mask to obtain a fusion information tensor;

[0017] Based on the fusion information tensor, predicting a three-dimensional space deformation field through a deformable registration network to perform a spatial transformation on the moving image, so that the moving image is aligned with the target image after deformation.

[0018] In a second aspect, the present application provides a multi-modal medical image registration system based on a segmentation large model, comprising:

[0019] An acquisition module configured to acquire a preoperative prostate three-dimensional image and an intraoperative prostate three-dimensional image as a pair of images to be registered;

[0020] A registration module configured to input the pair of images to be registered into the trained registration model to obtain a registration result of the pair of images to be registered;

[0021] The processing procedure of the registration model on the pair of images to be registered is as follows:

[0022] The preoperative prostate three-dimensional image is taken as a moving image, and the intraoperative prostate three-dimensional image is taken as a target image;

[0023] The segmentation large model is used to extract a moving mask and a target mask from the moving image and the target image respectively;

[0024] The moving image, the target image, the moving mask and the target mask are fused to obtain a fusion information tensor;

[0025] Based on the fusion information tensor, a three-dimensional space deformation field is predicted through a deformable registration network to perform a spatial transformation on the moving image, so that the moving image is aligned with the target image after deformation.

[0026] In a third aspect, the present application provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are run by the processor, the method of the first aspect is completed.

[0027] In a fourth aspect, the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by a processor, the method of the first aspect is completed.

[0028] The above one or more technical solutions have the following beneficial effects:

[0029] The present application fuses the original image with the segmentation mask information, integrates the segmentation details into the image features, effectively combines the image intensity and the anatomical boundary information, and improves the registration accuracy; based on the fusion information tensor, a three-dimensional space deformation field is predicted to realize the spatial transformation of the moving image, and the deformed moving image is highly consistent with the target image in the anatomical structure position and morphology, while ensuring the smoothness and continuity of the displacement field, meeting the accuracy and reliability requirements of multi-modal image registration in clinical practice, and providing an accurate image fusion basis for medical applications such as prostate surgery.

[0030] The pre-training capability of the segmentation module integrated SAM is only needed to fine-tune a small amount of samples to realize high-precision segmentation of the prostate region. This weakly supervised strategy based on the basic model greatly reduces the demand for manual labeling, reduces the data acquisition and preprocessing cost in clinical application, and avoids the registration deviation caused by labeling errors.

[0031] Advantages of the additional aspects of the application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0032] The accompanying drawings, which form a part of this description, are included to provide further understanding of the application, and are incorporated in and constitute a part of this description. The illustrative embodiments of the application and their description serve to explain the application. They do not, however, limit the application, nor do they express all of its novel features.

[0033] Figure 1 A flow chart of a multi-modal medical image registration method based on a segmentation large model in embodiment one of the application;

[0034] Figure 2 A three-dimensional voxel segmentation large model network structure diagram based on SAM in embodiment one of the application;

[0035] Figure 3 An information fusion layer network structure diagram in embodiment one of the application;

[0036] Figure 4 A registration network layer network structure diagram in embodiment one of the application. DETAILED DESCRIPTION

[0037] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the application. Unless otherwise indicated, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the application pertains.

[0038] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the application.

[0039] The embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0040] Embodiment one

[0041] The embodiment discloses a multi-modal medical image registration method based on a segmentation large model, comprising:

[0042] Obtaining a preoperative prostate three-dimensional image and an intraoperative prostate three-dimensional image as a pair of images to be registered;

[0043] inputting the to-be-registered image pair into the trained registration model to obtain a registration result of the to-be-registered image pair;

[0044] The registration model processes the to-be-registered image pair as follows:

[0045] The preoperative prostate three-dimensional image is taken as the moving image, and the intraoperative prostate three-dimensional image is taken as the target image.

[0046] The segmentation large model is used to extract a moving mask and a target mask from the moving image and the target image respectively.

[0047] The moving image, the target image, the moving mask and the target mask are fused to obtain a fusion information tensor.

[0048] Based on the fusion information tensor, a three-dimensional space deformation field is predicted through a deformable registration network to perform spatial transformation on the moving image, so that the moving image is aligned with the target image after deformation.

[0049] In this embodiment, the original image and the segmentation mask information are fused, the segmentation details are integrated into the image features, the image intensity and the anatomical boundary information are effectively combined, and the registration accuracy is improved. Based on the fusion information tensor, a three-dimensional space deformation field is predicted to realize spatial transformation of the moving image. The deformed moving image is highly consistent with the target image in terms of anatomical structure position and morphology, while ensuring smooth and continuous displacement field, meeting the accuracy and reliability requirements of clinical multi-modal image registration, and providing accurate image fusion basis for medical applications such as prostate surgery.

[0050] The following will be described in detail Figure 1 A multi-modal medical image registration method based on a segmentation large model is described in detail.

[0051] The registration model of the segmentation-fusion-registration integrated architecture is used for medical image registration. The registration model includes a SAM-based three-dimensional voxel segmentation large model, an information fusion layer, and a registration network layer.

[0052] S1: Obtain a preoperative prostate three-dimensional image as a moving image , obtain an intraoperative prostate three-dimensional image as a target image , and construct a to-be-registered image pair.

[0053] In S1, a nuclear magnetic resonance imaging instrument, an ultrasonic imaging instrument, etc. can be selected by those skilled in the art to obtain the preoperative and intraoperative prostate three-dimensional images of the patient.

[0054] S2: Use a SAM-based three-dimensional voxel segmentation large model to extract a moving mask and a target mask , accurately define important segmentation structures such as lesion regions.

[0055] As shown in the SAM-based three-dimensional voxel segmentation large model, the moving image and the target image are input, and the segmentation mask is predicted as output, which specifically includes: Figure 2

[0056] S201: input the moving image and the target image , respectively, into the normalization layer to standardize the pixel intensity of the image and eliminate the brightness difference between different modalities;

[0057] S202: capture the long-distance spatial dependence relationship of the moving image and the target image across the slices through the memory attention layer, enhance the modeling ability of the prostate region context information, and suppress irrelevant background interference;

[0058] S203: perform spatial dimension convolution operation on the features output by the memory attention layer through the three-dimensional convolution layer, extract three-dimensional features containing spatial and depth information, and capture the three-dimensional structure details of the prostate;

[0059] S204: after three-dimensional convolution, the high-dimensional features are compressed through the linear adjustment layer to reduce the channel dimension, retain key information, reduce computational complexity, control the number of new parameters, and improve model efficiency;

[0060] S205: the compressed features are input into the SAM module, and the pre-trained SAM model is used to complete the semantic segmentation of important structures with only a small amount of labeled samples, and generate a coarse-grained mask;

[0061] S206: the feature map is upsampled through the memory decoding layer to restore to the input image resolution, and output high-precision moving mask and target mask .

[0062] S3: through the information fusion layer, the moving image , the target image and their corresponding mask information are fused to output the information fusion result.

[0063] As shown in the SAM-based three-dimensional voxel segmentation large model, the moving image and the target image are input, and the segmentation mask is predicted as output, which specifically includes: Figure 3

[0064] S301: information direct splicing, the technical principle is to directly splice the moving image, the target image and their segmentation mask along the channel dimension to form a combined tensor containing original pixel intensity and anatomical boundary information.

[0065] The moving image ​​Target image Motion mask Target mask By directly splicing along the channel dimension, a combined tensor is generated. ,in, This means directly concatenating elements along the first dimension of the vector. This strategy directly preserves the original form of the image pixel intensity and the masked binary region. It is suitable for scenarios where there are large differences in image modalities and it is necessary to make equal use of intensity information and anatomical boundary information. For example, when the grayscale distribution of prostate MR images and ultrasound images differs significantly due to different imaging principles, direct concatenation can preserve the original features of each modality and provide comprehensive input information for the registration network.

[0066] S302: Feature fusion stitching. The technical principle is to selectively enhance the image features of the prostate region through feature extraction and weighted modulation, thereby reducing computational complexity while preserving key anatomical structural information.

[0067] Using the U-Net encoder, layer-wise feature extraction is performed on the motion mask and the target mask respectively to obtain the corresponding image feature tensors. and The image features are weighted and modulated using the confidence values ​​of the masked feature vectors, and enhanced feature representations are generated through element-wise multiplication and information combination operations. ,in, This represents element-wise multiplication, selectively enhancing the image features of the prostate region through the spatial distribution of the mask (such as a binary mask for the prostate region). It reduces computational complexity while preserving key anatomical information, making it suitable for registration tasks that require highlighting local anatomical features.

[0068] This strategy is suitable for registration tasks that require highlighting local anatomical features, such as precise alignment of prostate tumor regions. It can highlight image details of tumor boundaries by weighting mask features, while reducing computational redundancy in non-critical areas.

[0069] S4: Using the fusion results from the registration network layer, a three-dimensional spatial deformation field is calculated and generated, so that the moving image is transformed into a transformed moving image after the deformation field spatial transformation.

[0070] like Figure 4 As shown, the specific steps are as follows:

[0071] S401: The input to the registration network layer is the fused information tensor output by the information fusion layer. By downsampling, the spatial resolution of the fused information tensor is reduced to half of its original size, reducing computation while expanding the receptive field of the neural network to capture global structural features in the image.

[0072] S402: Perform three-dimensional convolution operation on the downsampled feature tensor, use 3×3×3 convolution kernel to extract local structural information in spatial and channel dimensions, and gradually build deep feature representation through multi-layer convolution stacking to capture anatomical structural relationships at different scales;

[0073] S403: By upsampling operations (such as trilinear interpolation), the deep feature representation is restored to near the original resolution, and by combining skip connections with shallow detail features from the downsampling stage, a multi-scale displacement field is formed. , , , Balancing global deformation trends with local structural alignment accuracy;

[0074] S404: The multi-scale displacement field is averaged element-wise, and a Gaussian smoothing filter is used to eliminate noise and anomalous displacement values, generating a smooth and continuous three-dimensional spatial deformation field. ;

[0075] S405: Utilizing the final three-dimensional spatial deformation field For moving images A spatial transformation is performed, and the new position of each pixel in the target image coordinate system is calculated using the nearest neighbor interpolation algorithm to generate the transformed moving image. , This indicates a transformation operation.

[0076] S5: Calculate the similarity loss and label overlap loss between the transformed moving image and the target image, and calculate the smoothness loss based on the deformation field to train the registration model.

[0077] Repeat steps S1-S5 until the registered model converges and training is complete.

[0078] The specific process is as follows:

[0079] S501: Similarity Loss The calculation uses normalized cross-correlation (NCC) loss, and normalization is used to eliminate gray-level differences between modes. The formula is as follows:

[0080]

[0081] in, For image pixel index, and These represent the transformed target image and the mean pixel value of the target image, respectively; Indicates a moving image. Represents the target image; It represents a three-dimensional spatial deformation field.

[0082] S502: Tag Overlap Loss Calculate using the transformed moving mask and target mask The formula for calculating the Dice coefficient loss is as follows:

[0083]

[0084] in, and These represent the volume of the intersection of the two masks and the total volume, respectively, both of which are scalars; For moving masks; For target mask.

[0085] S503: Deformation field smoothness loss To avoid discontinuous anomalous displacements in the deformation field, a gradient norm penalty term is used to constrain its spatial smoothness. The formula is as follows:

[0086]

[0087] in, This represents the total number of pixels in the deformation field. Indicates the deformation field at the pixel The spatial gradient at a given location is used to suppress abrupt shifts in displacement between adjacent pixels through the L2 norm, ensuring that the deformation conforms to the continuity of human anatomy. This represents the square of the L2 norm.

[0088] S504: Obtain the total loss by weighted summation of the three losses. :

[0089]

[0090] in, , , These are the weighting coefficients. The parameters of the registration network are updated using the backpropagation algorithm, iteratively adjusting the deformation field prediction capability.

[0091] S505: Set the maximum number of convergence threshold iterations to 1000. Training will terminate when one of the following conditions is met:

[0092] 1. The Dice coefficient on the validation set no longer increases within 100 consecutive iterations and the loss value tends to stabilize;

[0093] 2. Reach the preset maximum number of training rounds to avoid overfitting.

[0094] S6: Apply the trained registration network to the test, repeat steps S1-S4 to obtain the transformed image, map it to the real physical environment, and provide visualization assistance for prostate surgery.

[0095] The registration of preoperative prostate MR images with intraoperative 3D ultrasound images is performed according to steps S1-S4. This registration process overlays the preoperative MR-reconstructed 3D prostate structure onto the intraoperative ultrasound image, and marks key anatomical structures such as the tumor and urethra. Simultaneously, the registration results compensate for displacement errors caused by factors such as respiration and compression of the prostate, thus providing visual assistance for prostate surgery.

[0096] This embodiment constructs image pairs to be registered by acquiring preoperative and intraoperative 3D images. After normalization, a high-precision mask is extracted through a segmentation module containing a memory attention layer, a 3D convolutional layer, and a SAM model. The information fusion layer processes image and mask information using direct information stitching or feature fusion stitching strategies. The registration network layer generates a 3D deformation field through downsampling, 3D convolution, upsampling, and Gaussian smoothing to transform the moving image space. The network is trained to convergence using similarity loss, label overlap loss, and deformation field smoothness loss. During testing, the registration results are mapped to the physical environment to assist surgery. The computer acquires images through a DICOM interface. After segmentation, fusion, and registration, the images are aligned with the surgical space using an optical positioning system, providing a fused view in a semi-transparent overlay manner to assist surgical navigation.

[0097] This embodiment constructs a fully automated framework from image acquisition to physical space mapping. Preoperatively, multimodal images are acquired via MRI and ultrasound, and then segmented, fused, and registered to generate a deformation field. Intraoperatively, optical positioning sensors are used to map the registered virtual structure onto the surgical robot's coordinate system, achieving real-time fusion display of preoperative MRI and intraoperative ultrasound. This framework supports integration with mainstream surgical navigation systems, providing dynamic anatomical guidance for procedures such as prostate biopsy and tumor ablation, compensating for displacement errors caused by physiological movements such as breathing and compression, and improving surgical precision.

[0098] Example 2

[0099] The purpose of this embodiment is to provide a multimodal medical image registration system based on a large segmentation model, including:

[0100] The acquisition module is configured to acquire preoperative and intraoperative three-dimensional images of the prostate as image pairs to be registered.

[0101] The registration module is configured to input the trained registration model into the image pair to be registered, and obtain the registration result of the image pair to be registered.

[0102] The registration model processes the image pairs to be registered as follows:

[0103] The preoperative three-dimensional image of the prostate was used as the moving image, and the intraoperative three-dimensional image of the prostate was used as the target image.

[0104] The motion mask and target mask are extracted from the moving image and target image respectively using a large segmentation model;

[0105] The moving image, the target image, the moving mask, and the target mask are fused to obtain the fused information tensor;

[0106] Based on the fused information tensor, a deformable registration network is used to predict the 3D spatial deformation field, and the moving image is spatially transformed so that the moving image is aligned with the target image after deformation.

[0107] In further embodiments, the following is also provided:

[0108] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor. When executed by the processor, the computer instructions perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0109] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0110] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0111] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0112] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0113] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0114] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0115] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0116] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0117] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0118] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A multimodal medical image registration method based on a large segmentation model, characterized in that, include: Preoperative and intraoperative 3D images of the prostate were acquired as image pairs to be registered. The image pairs to be registered are input into the trained registration model to obtain the registration results of the image pairs to be registered. The registration model processes the image pairs to be registered as follows: The preoperative three-dimensional image of the prostate was used as the moving image, and the intraoperative three-dimensional image of the prostate was used as the target image. The motion mask and target mask are extracted from the moving image and target image respectively using a large segmentation model, specifically as follows: The pixel intensity of the moving image and the target image is standardized. By capturing long-range spatial dependencies across slices in moving or target images through a memory attention layer, the ability to model contextual information in the prostate region is enhanced. By performing spatial dimension convolution operations on the output features of the memory attention layer through a three-dimensional convolutional layer, three-dimensional features containing spatial and depth information are extracted to capture the three-dimensional structural details of the prostate. The high-dimensional features after 3D convolution are compressed in channel dimension through a linear adjustment layer; The compressed features are input into the pre-trained SAM convolution-deconvolution module to obtain the motion mask corresponding to the moving image or the target mask corresponding to the target image. The moving image, the target image, the moving mask, and the target mask are fused to obtain the fused information tensor; Based on the fused information tensor, a deformable registration network is used to predict the 3D spatial deformation field, and the moving image is spatially transformed so that the moving image is aligned with the target image after deformation.

2. The multimodal medical image registration method based on a large segmentation model as described in claim 1, characterized in that, The moving image, target image, motion mask, and target mask are fused to obtain a fused information tensor, specifically: The moving image, target image, moving mask, and target mask are concatenated along the channel dimension to generate a fused information tensor.

3. The multimodal medical image registration method based on a large segmentation model as described in claim 1, characterized in that, The moving image, target image, motion mask, and target mask are fused to obtain a fused information tensor, specifically: The hierarchical features of the motion mask and the target mask are extracted separately to obtain the image feature tensors corresponding to the motion mask and the target mask; The motion mask and the target mask are element-wise multiplied with their respective image feature tensors and then combined to obtain the fused information tensor.

4. The multimodal medical image registration method based on a large segmentation model as described in claim 1, characterized in that, Based on the fused information tensor, a deformable registration network is used to predict the 3D spatial deformation field and perform spatial transformation on the moving image, specifically: By downsampling, the spatial resolution of the fused information tensor is reduced to half of its original size; Three-dimensional convolution operations are performed on the downsampled feature tensor to extract local structural information in spatial and channel dimensions. Through multi-layer convolution stacking, deep feature representations are gradually constructed to capture anatomical structural relationships at different scales. The deep feature representation is restored to the original resolution by upsampling, and the shallow detail features of the downsampling stage are fused by skip connections to form a multi-scale displacement field. The multi-scale displacement field is averaged element by element, and Gaussian smoothing filter is used to eliminate noise and abnormal displacement values, generating a smooth and continuous three-dimensional spatial deformation field. The moving image is spatially transformed using a three-dimensional spatial deformation field. The new position of each pixel in the target image coordinate system is calculated to generate the transformed moving image.

5. The multimodal medical image registration method based on a large segmentation model as described in claim 1, characterized in that, The loss function used to train the registration model includes similarity loss, label overlap loss, and deformation field smoothness loss.

6. A multimodal medical image registration method based on a large segmentation model as described in claim 1 or 5, characterized in that, The loss function for training the registration model is specifically as follows: ； ； ； ； in, For image pixel index, and These represent the transformed target image and the mean pixel value of the target image, respectively; Indicates a moving image. Represents the target image; Represents a three-dimensional spatial deformation field; For moving masks; For target mask; and These represent the volume of the intersection of the transformed moving mask and the target mask, and the total volume, respectively. Indicates the deformation field at the pixel Spatial gradient at a given location; The loss function for the registration model; For similarity loss; For label overlap loss; This represents the loss of smoothness in the deformation field; , , Here, represents the weighting coefficient, and N represents the total number of pixels in the deformation field; This represents the square of the L2 norm.

7. A multimodal medical image registration system based on a large segmentation model, characterized in that, include: The acquisition module is configured to acquire preoperative and intraoperative three-dimensional images of the prostate as image pairs to be registered. The registration module is configured to input the trained registration model into the image pair to be registered, and obtain the registration result of the image pair to be registered. The registration model processes the image pairs to be registered as follows: The preoperative three-dimensional image of the prostate was used as the moving image, and the intraoperative three-dimensional image of the prostate was used as the target image. The motion mask and target mask are extracted from the moving image and target image respectively using a large segmentation model, specifically as follows: The pixel intensity of the moving image and the target image is standardized. By capturing long-range spatial dependencies across slices in moving or target images through a memory attention layer, the ability to model contextual information in the prostate region is enhanced. By performing spatial dimension convolution operations on the output features of the memory attention layer through a three-dimensional convolutional layer, three-dimensional features containing spatial and depth information are extracted to capture the three-dimensional structural details of the prostate. The high-dimensional features after 3D convolution are compressed in channel dimension through a linear adjustment layer; The compressed features are input into the pre-trained SAM convolution-deconvolution module to obtain the motion mask corresponding to the moving image or the target mask corresponding to the target image. The moving image, the target image, the moving mask, and the target mask are fused to obtain the fused information tensor; Based on the fused information tensor, a deformable registration network is used to predict the 3D spatial deformation field, and the moving image is spatially transformed so that the moving image is aligned with the target image after deformation.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

Images