Multi-modal image automatic registration method for left atrial appendage occlusion surgery navigation
Automatic registration of CTA and CBCT images was achieved using the U-Net deep learning framework and the visual Transformer network, which solved the problem of large image registration errors in left atrial appendage occlusion surgery, improved surgical efficiency and safety, reduced radiation risks, and promoted the standardization and popularization of the surgery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- RENJI HOSPITAL AFFILIATED TO SHANGHAI JIAO TONG UNIV SCHOOL OF MEDICINE
- Filing Date
- 2026-05-08
- Publication Date
- 2026-06-05
AI Technical Summary
In existing technologies, the registration error between CTA and CBCT images during left atrial appendage occlusion surgery is large, relying on manual adjustment, making it difficult to achieve standardized and accurate image fusion, resulting in prolonged operation time, increased radiation, and a higher risk of complications.
We employ U-Net and Visual Transformer networks based on deep learning frameworks to achieve automatic registration of CTA and CBCT images. By fusing deep convolution and Visual Transformer to calculate the affine transformation matrix, we achieve 3D affine registration of the images and generate accurate fused images.
It enables rapid and accurate registration of CTA and CBCT images, reduces surgical difficulty, decreases radiation dose and complication risk, promotes surgical efficiency and standardization, and is suitable for the widespread application of interventional surgery for structural heart disease.
Smart Images
Figure CN122156263A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cardiac CT image processing technology, and in particular to a multimodal automatic image registration method for navigation of left atrial appendage occlusion surgery. Background Technology
[0002] Cardiovascular disease (CVD) is the leading cause of death among Chinese residents, and its incidence continues to rise. Interventional treatment of structural heart disease has become the fastest-growing field. Transcatheter left atrial appendage occlusion (LAAO), as an important means of stroke prevention in patients with non-valvular atrial fibrillation, is experiencing explosive growth. However, current LAAO procedures still heavily rely on the surgeon's experience and the technician's manual image adjustments, requiring repeated switching and fusion between preoperative cardiovascular CTA (CT angiography) and intraoperative cone-beam computed tomography (CBCT). This leads to non-standardized procedures, prolonged operation time, increased X-ray exposure, and a higher risk of complications, and significantly extends the learning curve, limiting the technology's widespread adoption in primary care hospitals.
[0003] In LAAO surgery, cross-modal registration between preoperative CTA and intraoperative CBCT is far more challenging than intramodal image registration. The two imaging principles differ fundamentally: CTA employs a fan-beam multi-row detector CT, offering high spatial resolution (sub-millimeter level), excellent soft tissue contrast, and clear contrast enhancement, clearly displaying the anatomical morphology of the left atrial appendage, thrombi, and surrounding structures; CBCT, on the other hand, uses cone-beam geometry acquisition, limited by the intraoperative C-arm equipment, resulting in severe scattering artifacts, low soft tissue contrast, high noise levels, a smaller effective field of view, and uneven resolution, leading to completely different image intensity distribution, contrast mechanisms, and noise characteristics. Traditional methods based on grayscale or mutual information fail in cross-modal scenarios, often resulting in registration accuracy errors on the order of millimeters, poor robustness, and difficulty meeting clinical navigation needs. While existing research has attempted deep learning feature alignment or adversarial networks, a dedicated solution for cardiac dynamic CTA-CBCT remains lacking, and standardized automatic registration technology urgently needs breakthroughs.
[0004] Due to the lack of mature, integrated AI solutions for cardiac dynamic characteristics, especially in the area of automatic registration of the entire LAAO process, most existing fusion methods rely on manual labeling or semi-automatic adjustment, which cannot completely get rid of experience dependence and have a low degree of standardization. Summary of the Invention
[0005] This invention aims to address the technical problem of large registration errors between CTA and CBCT images in current clinical left atrial appendage occlusion surgery, which still requires manual adjustment. It provides an automatic registration method for CBCT and CTA multimodal images based on a deep learning framework. This method enables automatic, rapid, and accurate 3D affine registration of CTA and CBCT multimodal images. The fused images after registration provide clearer and more precise three-dimensional cardiac structural information for interventional surgeries such as LAAO, reducing complications and radiation dose, lowering surgical difficulty, and facilitating the standardized promotion and application of new technologies in interventional surgery for structural heart disease.
[0006] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:
[0007] In one aspect of the present invention, an automatic multimodal image registration method is provided, comprising the following steps:
[0008] Acquire preoperative CTA images and intraoperative CBCT images of patients undergoing interventional surgery for structural heart disease, and standardize the image format; The U-Net deep learning framework was used to segment and extract the cardiac region from CTA and CBCT images, and the extracted cardiac region images were preprocessed. A network that fuses deep convolution and visual Transformer is used to calculate the affine transformation matrix for preprocessed CTA and CBCT cardiac region images, enabling 3D affine registration of the CTA and CBCT images. The registered image is filled into the cropped area of the CBCT image to generate an intraoperative CBCT image containing cardiac anatomical information.
[0009] Preferably, the structural heart disease interventional procedure includes transcatheter left atrial appendage occlusion.
[0010] Preferably, the image format is uniformly nii format, and both the acquired CTA images and CBCT images have cardiac region masking.
[0011] Preferably, the preprocessing of the extracted cardiac region image includes: image size normalization and image data enhancement.
[0012] In some embodiments of the present invention, the image size is normalized to 256 × 256 × 224 pixels.
[0013] Preferably, the U-Net deep learning framework includes an encoder and a decoder. The encoder extracts features progressively through convolutional layers and downsampling layers, and the decoder reconstructs spatial details through upsampling and skip connections. The skip connections directly fuse high-resolution encoder features with the decoder output.
[0014] Preferably, the network that fuses depthwise convolution and visual Transformer calculates the affine transformation matrix for the preprocessed CTA and CBCT cardiac region images, enabling 3D affine registration between the CTA and CBCT images, including the following steps: Let the preprocessed CBCT image be a fixed image F, the CTA image be a moving image M, and the predicted affine transformation matrix be A; The predicted affine transformation matrix A is applied to the moving image M to generate the aligned and corrected image. To achieve 3D affine registration; The fixed image F and the moving image M use the same deep convolutional backbone network to extract features and project the images onto a fixed-dimensional feature space. The visual Transformer comprises a multi-head self-attention Transformer encoder and a parameter prediction head. The multi-head self-attention Transformer encoder consists of multiple consecutive encoder modules, each employing standard multi-head self-attention, feedforward networks, residual connections, and layer normalization. The parameter prediction head is a lightweight, multi-layer, fully connected neural network that directly regresses affine parameters from the encoded global feature vector. These parameters correspond to three-dimensional translation, Euler angle rotation, axial scaling, and shearing components, respectively, and are then combined to obtain a single affine transformation matrix A.
[0015] Preferably, the step of registering the CTA image with the CBCT image is implemented using a CBCT-CTA image registration model. This CBCT-CTA image registration model is trained in a semi-supervised manner, and its loss function is... L Similarity loss function Weighted sum with Dice loss function:
[0016] Among them, the similarity loss function The unsupervised, locally normalized cross-correlation loss function NCC is adopted, which is expressed as follows:
[0017] In the formula, For normalized cross-correlation based on a local window, the window size is... The calculation formula is as follows:
[0018] In this formula, For fixed images, For the aligned moving image ; The Dice loss, as an auxiliary supervision term, is calculated using the following formula:
[0019] in The number of categories labeled in the mask. and The first and second images are respectively the fixed image and the transformed moving image. Class separator tag.
[0020] In another aspect of the present invention, a multimodal image automatic registration system is also provided, comprising: The image acquisition module is used to acquire preoperative CTA images and intraoperative CBCT images; The image ROI region extraction module is used to segment and extract ROI region images from the CTA images and CBCT images using the U-Net deep learning framework; The CTA and CBCT multimodal image registration module is used to calculate the affine transformation matrix through a network that fuses depth convolution and visual Transformer, thereby achieving 3D affine registration from CTA to CBCT.
[0021] In another aspect of the present invention, an apparatus for automatic registration of multimodal images is also provided, comprising: Memory, used to store programs; A processor is used to execute the program stored in the memory to implement the above-described automatic multimodal image registration method.
[0022] In another aspect of the present invention, a computer-readable storage medium is also provided, which stores a program that can be executed by a processor to implement the above-described multimodal image automatic registration method.
[0023] This invention presents an automatic registration method for preoperative CTA and intraoperative CBCT multimodal images used in left atrial appendage occlusion (LAAO) surgical navigation. This method achieves precise spatial alignment between high-resolution preoperative CTA and intraoperative CBCT, generating a fused image view that effectively overcomes the limitations of a single modality. The fused image provides surgeons with a more comprehensive three-dimensional anatomical reference, aiding in precise instrument positioning, surgical path planning, and occluder size selection, significantly improving the success rate of device implantation. Furthermore, the automatic registration process replaces the traditional manual registration, significantly shortening intraoperative image fusion time, optimizing the surgical procedure, reducing overall surgical time, and significantly reducing X-ray exposure dose for both patients and medical staff, thus minimizing long-term radiation-related risks. This method also lowers the surgical skill threshold, shortens the learning curve, and enables more primary care hospitals and younger surgeons to safely perform complex LAAO surgeries, promoting the standardization and widespread adoption of interventional treatment for structural heart disease, and has broad application prospects. Attached Figure Description
[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0025] Figure 1 This is a flowchart of the automatic registration method for CTA and CBCT multimodal images based on a deep learning framework, according to the present invention.
[0026] Figure 2 This is a schematic diagram of the functional modules of the multimodal image automatic registration system of the present invention;
[0027] Figure 3 This is a schematic diagram of the cross-modal registration results of LAAO surgery according to an embodiment of the present invention. Detailed Implementation
[0028] Interventional treatment of structural heart disease, especially transcatheter left atrial appendage occlusion (LAAO), has become an important means of stroke prevention in patients with non-valvular atrial fibrillation. However, the preoperative cardiovascular CTA and intraoperative cone-beam computed tomography (CBCT) images have significant modal differences, making it difficult to achieve efficient and accurate spatial alignment in automatic registration. To solve this technical problem, this invention provides an automatic registration method for CTA and CBCT multimodal images based on a deep learning framework (see...). Figure 1 The method includes the following steps: acquiring preoperative CTA images and intraoperative CBCT images of patients undergoing interventional surgery for structural heart disease, and standardizing the image format; using the U-Net deep learning framework to segment and extract the cardiac region from the CTA and CBCT images, and preprocessing the extracted cardiac region images; calculating the affine transformation matrix of the preprocessed CTA and CBCT cardiac region images using a network fused with deep convolution and visual Transformer, enabling 3D affine registration of the CTA and CBCT images; filling the cropped area of the CBCT image with the registered image to generate an intraoperative CBCT image containing cardiac anatomical information. This method achieves automatic, rapid, and accurate spatial registration of preoperative cardiovascular CTA and intraoperative CBCT images. The fused image view generated by the registration results provides the surgeon with clear and accurate three-dimensional cardiac structural information and a comprehensive reference for real-time instrument position, effectively assisting in instrument positioning, path planning, and size selection, providing a new intelligent tool for navigation in interventional surgery for structural heart disease.
[0029] This invention's multimodal image automatic registration method precisely aligns complementary information from different imaging modalities (CTA providing high-resolution preoperative 3D anatomy and vascular enhancement information, and CBCT providing real-time intraoperative instrument placement and chamber structure) spatially, offering a comprehensive perspective for clinical decision-making and surgical assistance. Furthermore, this method focuses on efficient automatic registration during surgery (completing key steps within seconds), effectively replacing manual adjustments, improving surgical efficiency and safety, and reducing X-ray exposure time for both patients and medical staff. This method is expected to significantly improve the accuracy of interventional procedures for structural heart diseases such as LAAO, reduce complications and radiation dose, lower surgical difficulty, and enable more hospitals to perform complex interventional procedures.
[0030] In some embodiments of the present invention, an automatic registration method for CTA and CBCT multimodal images based on a deep learning framework is used for navigation in transcatheter left atrial appendage occlusion surgery, specifically including the following steps: Step 1: Acquire preoperative CTA images and intraoperative CBCT images of patients undergoing transcatheter left atrial appendage occlusion surgery, and convert the original dicom format images to nii format; Step 2, Image ROI region extraction and data augmentation: The heart region of CTA and CBCT images is separated and preserved using the U-Net network. Subsequently, the image size is normalized to 256 × 256 × 224 pixels to facilitate subsequent network input. At the same time, data augmentation techniques are used to improve image clarity. The third step involves using a network that fuses deep convolution and visual Transformer to calculate the affine transformation matrix for the normalized CTA (moving image) and CBCT (stationary image) images. The predicted affine transformation matrix is then applied to the moving image to generate an aligned, warped image. This enables fast and accurate 3D affine registration. Step 4: Align and register the corrected images The size is adjusted to the original cropped size and filled into the cropped area in the CBCT image to obtain an intraoperative CBCT image with cardiac anatomical information for intraoperative 3D navigation.
[0031] In some embodiments of the present invention, the second step of image ROI region extraction and data augmentation considers that the heart region is only distributed in local areas of CBCT and CTA images. Therefore, the heart region on the image needs to be segmented and cropped first, and then image registration is performed. This step is implemented through a heart ROI extraction model, which uses the U-Net deep learning framework for heart region segmentation and extraction. The U-Net deep learning framework is an encoder-decoder architecture for image segmentation, employing a symmetrical U-shaped design. It includes an encoder and a decoder. The encoder progressively extracts features through convolutional layers and downsampling layers, while the decoder reconstructs spatial details through upsampling and skip connections. These skip connections directly fuse high-resolution encoder features with the decoder output. The fully convolutional nature of the U-Net deep learning framework allows it to process input images of arbitrary size without resizing, while the skip connections span the semantic and spatial information of the deep network connections, thereby accurately locating boundaries.
[0032] In some embodiments of this invention, the cardiac ROI extraction model is trained on CBCT and CTA image datasets collected by this invention, with each dataset containing cardiac region mask annotations. The sample size ratio of the training set, validation set, and test set is 8:1:1. During the training phase, the model is trained for 50 epochs with a batch size of 32. During the testing phase, the trained cardiac ROI extraction model can automatically crop the cardiac region from the image based on the predicted cardiac region. All models are implemented using the PyTorch framework.
[0033] In some embodiments of the present invention, the third step, which uses a network fused with depthwise convolution and visual Transformer to calculate the affine transformation matrix for normalized CTA and CBCT cardiac region images, to achieve 3D affine registration between the CTA and CBCT images, includes the following steps: Let the normalized CBCT image be the fixed image F and the CTA image be the moving image M. Using the fixed image F and the moving image M as input, predict the affine transformation matrix A. The predicted affine transformation matrix A is applied to the moving image M to generate the aligned and corrected image. To achieve 3D affine registration.
[0034] In this process, the fixed image F and the moving image M use the same deep convolutional backbone network to extract features and project the images onto a fixed-dimensional feature space.
[0035] The Visual Transformer consists of a multi-head self-attention Transformer encoder and a parameter prediction head. The multi-head self-attention Transformer encoder comprises four consecutive encoder modules, each employing standard multi-head self-attention, feedforward networks, residual connections, and layer normalization. The self-attention mechanism and global receptive field of the Transformer encoder modules enable the network to simultaneously focus on distant anatomical structures in the image, effectively capturing large initial misalignments.
[0036] The parameter prediction head is a lightweight, multi-layer, fully connected neural network that directly regresses 12 affine parameters from the encoded global feature vector. These 12 parameters correspond to three-dimensional translation, Euler angle rotation, axial scaling, and shearing components, respectively, and are then combined to obtain a single affine transformation matrix A.
[0037] In some embodiments of the present invention, the third step of registering CTA images with CBCT images is implemented using a CBCT-CTA image registration model. This CBCT-CTA image registration model is trained in a semi-supervised manner, and its loss function is... L Similarity loss function Weighted sum with Dice loss function:
[0038] Among them, the similarity loss function The unsupervised, locally normalized cross-correlation loss function NCC is adopted, which is expressed as follows:
[0039] In the formula, For normalized cross-correlation based on a local window, the window size is given by the following formula:
[0040] In this formula, For fixed image , For the aligned moving image ; The Dice loss, as an auxiliary supervision term, is calculated using the following formula:
[0041] in This represents the number of categories labeled in the mask, which is 1 here. The first-class segmentation labels are for the fixed image and the transformed, moved image, respectively.
[0042] The CBCT and CTA image registration model was trained in an end-to-end manner using the Adam optimizer.
[0043] In some embodiments of the present invention, a multimodal image automatic registration system ( Figure 2 ),include: The image acquisition module is used to acquire preoperative CTA images and intraoperative CBCT images; The image ROI region extraction module segments and extracts ROI region images from the CTA and CBCT images using the U-Net deep learning framework; The CTA and CBCT multimodal image registration module is used to calculate the affine transformation matrix through a network that fuses depth convolution and visual Transformer, thereby achieving 3D affine registration from CTA to CBCT.
[0044] In some embodiments of the present invention, an apparatus for automatic registration of multimodal images includes: a memory for storing a program; and a processor for executing the program stored in the memory to implement an automatic registration method for CTA and CBCT multimodal images based on a deep learning framework.
[0045] The device includes a high-performance computing server, preferably a dedicated server for deep learning training and inference. This server can be deployed in a local data center or a cloud computing platform, supporting massively parallel computing and GPU acceleration.
[0046] The server can interact with users or upper-layer application systems through network interfaces and remote management tools (such as SSH, Web console, REST API, etc.), supporting multi-user remote access and task scheduling.
[0047] The memory includes at least one type of readable storage medium, including but not limited to: solid-state drives (SSDs), enterprise-class hard drives (HDDs), caches (such as NVMe storage), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), etc. In some embodiments, the memory may include the server's internal storage units, such as system memory (DDR4 / DDR5 memory), GPU memory (VRAM), and local high-speed storage arrays. In other embodiments, the memory may also include external or distributed storage devices, such as network-attached storage (NAS), storage area networks (SANs), object storage systems, or cloud storage services. The memory may simultaneously contain both the server's local storage units and external / distributed storage systems.
[0048] In this embodiment of the invention, the memory is used to store deep learning frameworks (such as PyTorch, TensorFlow, etc.), pre-trained model weights, medical image datasets, training scripts, and configuration files. Furthermore, the memory can also be used to temporarily store intermediate results generated during training, model checkpoints, batch image data, and inference output results.
[0049] In some embodiments, the processor may be a central processing unit (CPU), graphics processing unit (GPU), tensor processor (TPU), or other artificial intelligence acceleration chips (such as NVIDIA A100 / H100 series, AMD Instinct series, Ascend series, etc.). The processor typically works in conjunction with multiple high-performance accelerator cards (such as multi-GPUs) to control the overall computation of the server. In this embodiment of the invention, the processor and accelerator cards are used to run deep learning program code stored in memory, perform training and inference tasks of neural network models, and process large-scale medical image data.
[0050] In some embodiments of the present invention, a computer-readable storage medium stores a program that can be executed by a processor to implement the above-described multimodal image automatic registration method.
[0051] Example 1 1. Experimental Data This experiment used intraoperative CBCT and preoperative CTA images collected clinically from 57 patients with LAAO. All 114 images were annotated with masking information. 51 samples were randomly selected for training and validation, and the remaining 6 samples were used for testing. All data and masking annotations were verified by professional physicians.
[0052] 2. Experimental Results This invention was tested on a collected test set of LAAO patients. The results show that the method of this invention can accurately achieve cross-modal registration of intraoperative CBCT images and preoperative CTA images, supplementing CBCT images with rich cardiac structural information. Statistical results on the test set show that the method of this invention achieves an average Dice value of 0.92±0.04 in the cardiac region and 0.87±0.03 in the left atrium and left atrial appendage regions, significantly better than the results of direct registration of the original images (average Dice values of 0.83±0.02 and 0.75±0.03, respectively) and the results achieved by traditional registration methods. Partial test set results are shown below. Figure 3 As shown. Figure 3The first row of images shows preoperative CTA and intraoperative CBCT images. The acquisition time, imaging equipment, and field of view differ significantly between the two, resulting in substantial differences in the information contained within them. Preoperative CTA images have high resolution, clearly showing the anatomical structure of the heart, while intraoperative CBCT images have lower resolution but provide information about the patient's intraoperative position. The second row of images shows CTA (moving image) and CBCT (fixed image) images after extraction and normalization of the cardiac region of interest (ROI). The left image in the third row is a fused CBCT image registered using the method of this invention, containing aligned cardiac anatomical information; the right image is the final CBCT image, which can be used for subsequent 3D reconstruction and surgical navigation. The entire left atrial appendage occlusion image registration and fusion process took an average of 4.2 ± 0.7 seconds on a standard medical workstation (equipped with an NVIDIA RTX 4090D GPU), which is much shorter than the approximately 285 seconds required by traditional iterative optimization registration methods and also better than the time required for manual registration by professional doctors. This greatly reduced the waiting time for image processing during surgery, alleviated the difficulty of the surgery, and improved the efficiency of the surgery.
[0053] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the above functions are implemented by executing the program with a computer. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be implemented. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash drive, or portable hard drive, and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be implemented.
[0054] The embodiments described above are merely illustrative of implementation methods of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A method for automatic registration of multimodal images, characterized in that, Includes the following steps: Acquire preoperative CTA images and intraoperative CBCT images of patients undergoing interventional surgery for structural heart disease, and standardize the image format; The U-Net deep learning framework was used to segment and extract the cardiac region from CTA and CBCT images, and the extracted cardiac region images were preprocessed. A network that fuses deep convolution and visual Transformer is used to calculate the affine transformation matrix for preprocessed CTA and CBCT cardiac region images, enabling 3D affine registration of the CTA and CBCT images. The registered image is filled into the cropped area of the CBCT image to generate an intraoperative CBCT image containing cardiac anatomical information.
2. The multimodal image automatic registration method according to claim 1, characterized in that, The structural heart disease interventional procedures include transcatheter left atrial appendage occlusion.
3. The multimodal image automatic registration method according to claim 1, characterized in that, The image format is uniformly nii format, and both the acquired CTA and CBCT images have cardiac region masking.
4. The multimodal image automatic registration method according to claim 1, characterized in that, The preprocessing of the extracted cardiac region image includes: image size normalization and image data augmentation.
5. The multimodal image automatic registration method according to claim 1, characterized in that, The U-Net deep learning framework includes an encoder and a decoder. The encoder extracts features stepwise through convolutional layers and downsampling layers, while the decoder reconstructs spatial details through upsampling and skip connections. The skip connections directly fuse high-resolution encoder features with the decoder output.
6. The multimodal image automatic registration method according to claim 1, characterized in that, The network, which fuses depthwise convolution and visual Transformer, calculates the affine transformation matrix for the preprocessed CTA and CBCT cardiac region images, enabling 3D affine registration between the CTA and CBCT images. This includes the following steps: Let the preprocessed CBCT image be a fixed image F, the CTA image be a moving image M, and the predicted affine transformation matrix be A; The predicted affine transformation matrix A is applied to the moving image M to generate the aligned and corrected image. To achieve 3D affine registration; The fixed image F and the moving image M use the same deep convolutional backbone network to extract features and project the images onto a fixed-dimensional feature space. The visual Transformer comprises a multi-head self-attention Transformer encoder and a parameter prediction head. The multi-head self-attention Transformer encoder consists of multiple consecutive encoder modules, each employing standard multi-head self-attention, feedforward networks, residual connections, and layer normalization. The parameter prediction head is a lightweight, multi-layer, fully connected neural network that directly regresses affine parameters from the encoded global feature vector. These parameters correspond to three-dimensional translation, Euler angle rotation, axial scaling, and shearing components, respectively, and are then combined to obtain a single affine transformation matrix A.
7. The multimodal image automatic registration method according to claim 6, characterized in that, The step of registering CTA images with CBCT images is implemented using a CBCT-CTA image registration model. This CBCT-CTA image registration model is trained in a semi-supervised manner, and its loss function is... L Similarity loss function Weighted sum with Dice loss function: Among them, the similarity loss function The unsupervised, locally normalized cross-correlation loss function NCC is adopted, which is expressed as follows: In the formula, For normalized cross-correlation based on a local window, the window size is... The calculation formula is as follows: In this formula, a fixed image is used. , For the aligned moving image ; The Dice loss, as an auxiliary supervision term, is calculated using the following formula: in The number of categories labeled in the mask. and The first-class segmentation labels are for the fixed image and the transformed, moved image, respectively.
8. A multimodal image automatic registration system, characterized in that, include: The image acquisition module is used to acquire preoperative CTA images and intraoperative CBCT images; The image ROI region extraction module is used to segment and extract ROI region images from the CTA images and CBCT images using the U-Net deep learning framework; The CTA and CBCT multimodal image registration module is used to calculate the affine transformation matrix through a network that fuses depth convolution and visual Transformer, thereby achieving 3D affine registration from CTA to CBCT.
9. A device for automatic registration of multimodal images, characterized in that, include: Memory, used to store programs; A processor for executing a program stored in the memory to implement the multimodal image automatic registration method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a program that can be processed by a processor. Execute to implement the multimodal image automatic registration method as described in any one of claims 1-7.