Multi-modal medical image registration method, device and surgical robot navigation system
By employing a structure-aware Transformer regression model and an unsupervised training strategy guided by 3D contour maps, the problem of low accuracy in registering intraoperative ultrasound images to preoperative images was solved, achieving a more efficient non-rigid registration effect.
Patent Information
- Application Number
- CN202610785128.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-02
- Publication Date
- 2026-08-25
AI Technical Summary
Existing methods for registering intraoperative ultrasound images to preoperative images suffer from low registration accuracy, especially when dealing with images of different modalities and dimensions. Ultrasound images have low contrast and are noisy, and patient movement and surgical instrument interference can cause tissue deformation, making it difficult for existing unsupervised methods to achieve accurate registration.
Non-rigid registration is performed using a structure-aware Transformer regression model. A non-rigid registration coordinate map is generated by combining a position-aware image patch encoding and feature extraction and regression prediction module with an unsupervised training strategy guided by a 3D contour map and using self-attention and cross-modal attention mechanisms for feature fusion.
It significantly improves the registration accuracy of intraoperative ultrasound images to preoperative images, reduces computational load and memory consumption, overcomes the problems of modal differences and insufficient image information in traditional methods, and achieves more efficient non-rigid registration.
Smart Images
Figure CN122636680A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of medical image processing technology, and specifically relates to multi-modal medical image registration methods, equipment and surgical robot navigation systems. Background Technology
[0002] Intraoperative ultrasound image to preoperative image (e.g., CT / MR) 2D-3D registration technology is one of the core technologies in the autonomous navigation robot system of minimally invasive surgery. It visualizes the scanning position of intraoperative ultrasound in the three-dimensional space of the human body in real time, thereby expanding the surgeon's surgical field of vision and helping to locate lesions in real time.
[0003] Existing methods for registering intraoperative ultrasound images to preoperative images can be divided into vision-based (or tracking-free) methods and external sensor-based methods. Vision-based methods only require input of two-dimensional structural images and three-dimensional CT / MR volumetric data for registration, while external sensor-based methods require the introduction of additional auxiliary information. Vision-based methods are widely used due to their simple configuration and lack of additional surgical costs.
[0004] However, existing vision-based methods still face many challenges. First, the registration process involves registering images of different modalities and dimensions. Second, ultrasound images typically have low contrast and contain a lot of noise and artifacts, leaving only limited information for registration. In addition, the patient's own respiratory movements and the intervention of surgical instruments can cause tissue deformation, resulting in a mismatch between the two-dimensional structural images and the anatomical structures of tissues and organs in the three-dimensional structural data.
[0005] With the development of deep learning, intraoperative ultrasound image-to-preoperative image registration technology based on deep learning has been widely used. However, due to the difficulty in obtaining the ground truth labels for intraoperative ultrasound image-to-preoperative image registration caused by factors such as electromagnetic field disturbances, coordinate system registration errors, and human motion, supervised learning is difficult to achieve. Therefore, unsupervised registration methods have been widely discussed. Existing unsupervised methods can be divided into two categories: one is to generate a large number of synthetic virtual images for supervised training; the other is to use the image similarity between the input image and the synthetic image generated by the registration result for self-supervision. However, both types of unsupervised methods are difficult to achieve good registration results due to modal differences and limited image information. This is because the first type of method lacks learning of the structural features of real images during model training, while the second type of method, relying only on limited two-dimensional image similarity information, is insufficient for accurate optimization of the registration position.
[0006] In summary, existing methods for registering intraoperative ultrasound images to preoperative images suffer from low registration accuracy. Summary of the Invention
[0007] To address the aforementioned issues, this application provides a multi-modal medical image registration method, equipment, and surgical robot navigation system to improve the accuracy of intraoperative ultrasound image registration to preoperative image registration.
[0008] This application provides a multimodal medical image registration method, including: Pre-processing was performed on preoperative 3D images and intraoperative 2D ultrasound images to obtain 3D structural data and 2D structural image sequences. The three-dimensional structure data and the two-dimensional structure image sequence are input into a pre-trained structure-aware Transformer regression model; The input data is processed using a trained structure-aware Transformer regression model to output a non-rigid registration coordinate map, thereby achieving spatial registration from intraoperative two-dimensional ultrasound images to preoperative three-dimensional images. The structure-aware Transformer regression model includes a location-aware image patch encoding module and a feature extraction and regression prediction module. The position-aware image block encoding module is used to encode and remove redundant information from the input three-dimensional structure data and two-dimensional structure image sequence based on spatial and temporal position priors, and to map the processed image blocks into three-dimensional data tokens and two-dimensional data tokens. The feature extraction and regression prediction module is used to perform self-attention feature extraction on 3D and 2D data tokens respectively to obtain 3D global features and 2D local features. The 3D global features are fused into the 2D local features through a unidirectional cross-modal attention mechanism to obtain enhanced 2D features. Based on the enhanced 2D features, spatial coordinate regression is performed to generate a non-rigid registration coordinate map.
[0009] Further, the position-aware image block encoding module is specifically used to generate a 3D position map for marking the spatial position of voxels in the 3D structure data, and to generate a 2D position map for marking pixel coordinates and time sequence for each frame in the 2D structure image sequence; the 3D position map and the 3D structure data are concatenated by channels to obtain 3D enhancement features, and the 2D position map and the corresponding 2D structure image are concatenated by channels to obtain a 2D enhancement sequence; the 3D enhancement data and the 2D enhancement sequence are respectively encoded into 3D image blocks and 2D image blocks according to preset sizes to obtain 3D image blocks and 2D image blocks; invalid image blocks are removed from the 3D image blocks and the image blocks of all frames in the 2D enhancement sequence except the last frame, and all image blocks of the last 2D image are retained to obtain a set of 3D valid image blocks and a set of 2D valid image blocks; the set of 3D valid image blocks and the set of 2D valid image blocks are encoded by a multilayer perceptron to generate a 3D data token containing global spatial structure information and a 2D data token containing local temporal features, respectively.
[0010] Furthermore, the feature extraction and regression prediction module is specifically used to perform self-attention feature extraction on the 3D data token and the 2D data token using L1 and L2 Transformer layers respectively, to obtain 3D global features and 2D local features. Then, L3 Transformer layers are used to perform unidirectional cross-attention extraction, injecting the 3D global features into the 2D local features for cross-modal feature fusion, to obtain enhanced 2D features. The target data token corresponding to the last frame of the 2D image is extracted from the enhanced 2D features. The target feature token is then subjected to spatial dimension reshaping, upsampling, and convolutional mapping to generate a non-rigid registration coordinate map, where L1, L2, and L3 are positive integers greater than 1.
[0011] Furthermore, the training process of the structure-aware Transformer regression model is as follows: Structural segmentation is performed on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively. Based on the three-dimensional structure data, simulated scanning generates a sequence of synthesized two-dimensional structure images, and records the synthesized ground truth coordinate map; Based on three-dimensional structure data, a three-dimensional contour map with gradient decay characteristics is constructed according to the distance from the voxel to the nearest structure edge. The three-dimensional structure data and the synthesized two-dimensional structure image sequence are input into the structure-aware Transformer regression model, and the synthesized predicted coordinate map is output. The coordinate regression constraint loss is calculated based on the synthesized predicted coordinate map and the synthesized ground truth coordinate map. The three-dimensional structure data and the two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a true predicted coordinate map. Based on the true predicted coordinate map and the three-dimensional contour map, the contour line guided constraint loss is calculated. By combining coordinate regression constraint loss and contour-guided constraint loss, the structure-aware Transformer regression model is trained end-to-end in unsupervised manner.
[0012] Furthermore, the training process of the structure-aware Transformer regression model is as follows: Structural segmentation is performed on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively. Based on the three-dimensional structure data, simulated scanning generates a sequence of synthesized two-dimensional structure images, and records the synthesized ground truth coordinate map; The three-dimensional structure data and the synthesized two-dimensional structure image sequence are input into the structure-aware Transformer regression model, and the synthesized predicted coordinate map is output. The coordinate regression constraint loss is calculated based on the synthesized predicted coordinate map and the synthesized ground truth coordinate map. The three-dimensional structure data and the two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a true predicted coordinate map. A synthetic image is generated based on the true predicted coordinate map. The effective region constraint loss is calculated based on the two-dimensional structure image, the synthetic image, and the effective region mask of the two-dimensional structure image. By combining coordinate regression constraint loss and effective region constraint loss, end-to-end unsupervised training is performed on the structure-aware Transformer regression model.
[0013] Furthermore, the preoperative three-dimensional images and intraoperative two-dimensional ultrasound images were preprocessed to obtain three-dimensional structural data and two-dimensional structural image sequences, including: The system performs structural segmentation on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively.
[0014] This application provides an electronic device, including: Memory, which stores execution instructions; and The processor executes the instructions stored in memory, causing the processor to perform the methods described above.
[0015] This application also provides a surgical robot navigation system, including: the aforementioned electronic device.
[0016] This application also provides a readable storage medium storing executable instructions, which are executed by a processor to implement the above-described method.
[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0018] Compared with the prior art, this application has the following advantages: (1) The structure-aware Transformer regression model proposed in this application uses position-aware image block encoding to remove background blocks without information, thereby greatly reducing the amount of computation and memory consumption. At the same time, it can effectively perceive the global position of the remaining tokens, accurately extract features and perform non-rigid regression prediction, thereby improving the accuracy of registration.
[0019] (2) The joint synthetic real image unsupervised learning strategy proposed in this application, on the one hand, allows the model to learn basic registration and mapping capabilities through synthetic images, and on the other hand, allows the model to perceive the structural features of real images and the accurate optimization direction in three-dimensional space through contour line guided constraint loss, effectively overcoming the problems of weak supervision signals and ambiguous optimization directions in traditional effective region constraint loss.
[0020] (3) The three-dimensional contour map constructed in this application uses the exponential gradient decay mechanism to give different penalty weights to different spatial locations, so that the model can effectively perceive and converge to the correct anatomical structure edge position according to the loss value, which significantly improves the accuracy of non-rigid registration.
[0021] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0023] Figure 1 This is a schematic diagram of the structure-aware Transformer regression model provided in an embodiment of this application; Figure 2 A logical schematic diagram of the joint synthesis of real images unsupervised training strategy provided in the embodiments of this application; Figure 3 This is a framework diagram of an autonomous navigation system for a surgical vision robot provided in an embodiment of this application. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] Because two-dimensional structural images and three-dimensional preoperative volumetric imaging data have significant modal differences, and two-dimensional structural images contain a large amount of noise and artifacts that cannot be used for registration, the unsupervised non-rigid registration framework proposed in this application segments important anatomical structures (such as hepatic veins and portal veins in the liver) in intraoperative ultrasound and preoperative imaging. This is because hepatic vascular structures are widely distributed in the liver and can ensure structural consistency in intraoperative and preoperative images of different modalities.
[0026] In the registration task from 2D structural images to 3D preoperative volumetric images, deep learning models need to perform calculations on both 2D structural images and 3D preoperative volumetric images simultaneously, resulting in significant computational and memory consumption. Research has found that in structure-based registration tasks, both 2D structural images and 3D volumetric images contain numerous information-free (black background) regions. The visual Transformer's mechanism of segmenting images into blocks for feature extraction is well-suited for this task because it can remove these information-free image blocks, thus reducing the model's computational and memory requirements. However, traditional visual Transformers cannot effectively perceive the global location of remaining valid tokens for feature extraction. Therefore, the structure-aware Transformer regression model proposed in this application divides the input data into tokens using location-aware image block encoding, thereby accurately extracting global features and performing non-rigid regression prediction.
[0027] The multimodal medical image registration method of this application is as follows: Pre-processing was performed on preoperative 3D images and intraoperative 2D ultrasound images to obtain 3D structural data and 2D structural image sequences. The three-dimensional structure data and the two-dimensional structure image sequence are input into a pre-trained structure-aware Transformer regression model; The input data is processed using a trained structure-aware Transformer regression model to output a non-rigid registration coordinate map, thereby achieving spatial registration from intraoperative two-dimensional ultrasound images to preoperative three-dimensional images. The structure-aware Transformer regression model includes a location-aware image patch encoding module and a feature extraction and regression prediction module. The position-aware image block encoding module is used to encode and remove redundant information from the input three-dimensional structure data and two-dimensional structure image sequence based on spatial and temporal position priors, and to map the processed image blocks into three-dimensional data tokens and two-dimensional data tokens. The feature extraction and regression prediction module is used to perform self-attention feature extraction on the three-dimensional data token and the two-dimensional data token respectively to obtain three-dimensional global features and two-dimensional local features; the three-dimensional global features are fused into the two-dimensional local features through a one-way cross-modal attention mechanism to obtain enhanced two-dimensional features; and spatial coordinate regression is performed based on the enhanced two-dimensional features to generate the non-rigid registration coordinate map.
[0028] The following is combined Figure 1 The structure-aware Transformer regression model shown below provides a detailed explanation of the solution in this application's embodiments: Position-aware image block encoding is specifically used to generate a 3D position map for marking voxel spatial positions for the 3D structure data, and a 2D position map for marking pixel coordinates and time sequence for each frame in the 2D structure image sequence; the 3D position map and the 3D structure data are concatenated by channels to obtain 3D enhancement features, and the 2D position map and the corresponding 2D structure image are concatenated by channels to obtain a 2D enhancement sequence; the 3D enhancement data and the 2D enhancement sequence are respectively coded into 3D image blocks and 2D image blocks according to preset sizes to obtain 3D image blocks and 2D image blocks; invalid image blocks are removed from the 3D image blocks and all image blocks in the 2D enhancement sequence except for the last frame, and all image blocks in the last 2D image are retained to obtain a set of 3D valid image blocks and a set of 2D valid image blocks; the set of 3D valid image blocks and the set of 2D valid image blocks are encoded by a multilayer perceptron to generate a 3D data token containing global spatial structure information and a 2D data token containing local temporal features, respectively.
[0029] Specifically, the above process is described in mathematical terms as follows: For the input three-dimensional structure data With two-dimensional structure image sequence First, define a three-dimensional position map based on the input data dimensions. With two-dimensional location map
[0030]
[0031]
[0032] in, This indicates the number of voxels in the height direction of the 3D image. This indicates the number of voxels in the width direction of the 3D image. This indicates the number of voxels in the depth / slice direction of the 3D image; This indicates the number of voxels in the height direction of a 2D image. The number of voxels in the width direction of the 2D image is represented by T, which represents the sequence length; x represents the voxel coordinates in the width direction (horizontal axis); y represents the voxel coordinates in the height direction (vertical axis); and z represents the voxel coordinates in the depth / slice direction (which slice layer). Represents the pixel coordinates in the horizontal direction (width) of a single 2D image; Represents the pixel coordinates in the vertical direction (height) of a single 2D image; Total height in pixels of a single 2D image; This represents the total width in pixels of a single 2D image; Indicates the current slice number.
[0033] Then, define the location map , With the corresponding three-dimensional structure data , Perform channel splicing to obtain and After that, the 3D augmented data will be... With two-dimensional enhancement sequences Image blocks are obtained by dividing and encoding image blocks:
[0034]
[0035] in, Indicates the size of the image patch. and This indicates the number of 3D and 2D data image blocks.
[0036] Then, remove the 3D image blocks. With two-dimensional image patch set { The non-information blocks in the image are combined with the remaining valid information image blocks to obtain a set of three-dimensional valid image blocks. With two-dimensional effective image patch set , and This indicates the number of 3D and 2D data image blocks after removing blocks with no information.
[0037] Here, for the input two-dimensional image sequence, the model needs to retain all image patches of the last image for subsequent pixel-level regression coordinate map prediction, because in the application, only the last image of the input image sequence needs to be registered and predicted.
[0038] Finally, the three-dimensional effective image patch set is processed through two multilayer perceptron modules (MLP). With two-dimensional effective image patch set Encode to obtain 3D data tokens. With two-dimensional data tokens , This indicates the number of channels for the token.
[0039] The feature extraction and regression prediction modules are specifically used for... One and Each Transformer layer performs self-attention feature extraction on 3D and 2D data tokens to obtain 3D global features and 2D local features, which are then used... Each Transformer layer performs unidirectional cross-attention extraction, injecting 3D global features into 2D local features for cross-modal feature fusion to obtain enhanced 2D features. The target data token corresponding to the last frame of the 2D image is extracted from these enhanced 2D features. The target feature token is then subjected to spatial dimension reshaping, upsampling, and convolutional mapping to generate a non-rigid registration coordinate map. , and It is a positive integer greater than 1.
[0040] The above feature extraction and regression prediction process can be described mathematically as follows: After location-aware image block encoding and invalid information block removal, the model first uses... Each Transformer layer includes a multi-head self-attention mechanism (MSA) and a multilayer perceptron (MLP) to extract features from 3D data tokens to obtain 3D global features. and use another Each Transformer layer extracts features from the two-dimensional data Tokens to obtain two-dimensional local features. Then, to enable the 2D tokens to learn more comprehensive 3D spatial structural location information and thus predict more accurate registration results, the following method is used: Each Transformer layer performs unidirectional cross-attention extraction, injecting 3D global features into 2D local features. Perform cross-modal fusion to obtain enhanced two-dimensional features .
[0041] After feature extraction, the model first enhances the two-dimensional features. The target data token for the last image of the lieutenant general. The target data token contains sufficient spatial location information for predicting the registration results.
[0042] Then, the target data token of the last image is processed through a prediction head. Coordinate plots used to predict non-rigid registration :
[0043] in, This indicates a reshaping operation, which will target the data token. Transforming tokens into image-based spatial feature maps express A series of consecutive convolutional upsampling operations with a stride of 2. Finally, through a... The convolution yields a 3-channel registration result, which represents the registration of each pixel coordinate on the last 2D image to the coordinate position in the 3D structure data.
[0044] To address the problems of existing unsupervised image-to-3D structure data registration methods, this application proposes the following... Figure 2 The proposed strategy uses a joint synthetic real-image unsupervised learning approach to train a structure-aware Transformer regression model for non-rigid registration from 2D structural images to 3D structural data. First, it generates synthetic structural images and corresponding ground truth labels from the 3D structural data, training the model using a coordinate regression constraint loss. Second, the strategy constructs a 3D contour map from the 3D structural data and proposes a novel contour-guided constraint loss for self-supervised learning of the real input image. This joint synthetic and real-image unsupervised training effectively teaches the model how to perform non-rigid registration from 2D structural images to 3D structural data.
[0045] The specific training process is as follows: Structural segmentation is performed on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively. Based on the three-dimensional structure data, simulated scanning generates a sequence of synthesized two-dimensional structure images, and the synthesized ground truth coordinate map is recorded; Based on the three-dimensional structure data, a three-dimensional contour map with gradient decay characteristics is constructed according to the distance from the voxel to the nearest structural edge. The three-dimensional structure data and the synthesized two-dimensional structure image sequence are input into the structure-aware Transformer regression model, and the synthesized predicted coordinate map is output; the coordinate regression constraint loss is calculated based on the synthesized predicted coordinate map and the synthesized ground truth coordinate map. The three-dimensional structure data and the two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a true predicted coordinate map. Based on the true predicted coordinate map and the three-dimensional contour map, the contour line guided constraint loss is calculated. By combining the coordinate regression constraint loss and the contour line guided constraint loss, the structure-aware Transformer regression model is trained end-to-end without supervision.
[0046] The specific technical details involved are explained in detail below: Synthetic image generation: For the input 3D preoperative data, first, based on the voxel spacing... Resampling is performed, and then a pre-trained 3D segmentation model is used to segment the vascular structure of the input data to obtain 3D vascular structure data. The synthetic image generation algorithm fills in the image using image interpolation to obtain a two-dimensional structured image. :
[0047] in, (0, () indicates the position of the simulated ultrasound probe on the image. Represents the pixel spacing of a two-dimensional image. This indicates that the ultrasonic probe is in three-dimensional space. The position in the middle, and The plane that constitutes an ultrasound scan. By adjusting the position and angle of the scan, a large number of synthetic ultrasound structural image sequences can be obtained. Simultaneously, during the generation of the synthesized image, the ground truth coordinate map of the registration from the two-dimensional image to the three-dimensional structure data can be recorded. These are used as truth labels for model training.
[0048] Coordinate regression constraint loss: for input 3D structure data With two-dimensional synthetic structure image sequence First, the regression coordinate graph is predicted using the SATRN model. The coordinate regression constraint loss is then calculated as follows. :
[0049] Coordinate regression constraint loss can effectively supervise the model's registration from synthetic structural images to volume data. However, due to the significant modal differences between synthetic and real images, even though this application extracts vascular structures from the images using a segmentation model, directly applying the model trained on synthetic data using coordinate regression constraint loss to real structural images fails to achieve effective registration prediction. Therefore, this application proposes using contour-guided constraint loss calculated from real structural images to further train the model and improve registration accuracy.
[0050] Contour-guided constraint loss: Assumption It is the SATRN model for input three-dimensional structural data With two-dimensional structural images The predicted regression coordinate map can then be used to calculate a synthetic structure image. (The process is similar to that of synthesized image generation.) Traditional self-supervised learning strategies use image similarity metrics, such as the Structural Similarity Index Measure (SSIM), to calculate the input two-dimensional structural image. Synthetic image generated by registration The similarity is used as the training loss. However, since SATRN uses anatomical structural information for image-to-volume data registration, this application proposes a Valid-Region Constraint (VRC) loss for binary images that only contain structural information. The self-supervised method, replacing image similarity, is as follows:
[0051]
[0052] in, Representation of structure diagram A valid-region mask (VRM) is a two-dimensional structure diagram where positions with values are set to 1. "" indicates a pixel-wise multiplication operation. Effective region constraint loss allows the model to recognize that effective structural regions in a 2D structural image should be registered to effective regions in 3D structural data. However, the effective region constraint loss provides only a weak or ambiguous signal for model optimization because when a 2D structural point is not registered to any structure in the 3D data, the penalty for registering it to any other location is the same. Therefore, the supervision signal provided by this loss is insufficient for the model to determine the accurate optimization direction. This application proposes a novel contour-guided constraint loss to address this problem.
[0053] Specifically, firstly, based on the input three-dimensional structure data The structural information is used to define a three-dimensional contour map through exponentially decaying gradients. :
[0054] in, To control the magnitude of attenuation, Indicates the current voxel position The voxel distance to the nearest structural edge. Then, through this 3D contour plot. The above formula is redefined, thus proposing a novel contour-guided constraint loss. :
[0055]
[0056] With effective region constraint loss The difference lies in the loss constrained by contour lines. With supervision, the model can effectively perceive the correct optimization direction based on the loss value, thereby accurately registering the structural regions of the two-dimensional ultrasound image with those of the three-dimensional image.
[0057] The unsupervised training strategy for jointly synthesized real images proposed in this application combines coordinate regression constraint loss based on synthesized images. Compared with contour constraint loss based on real images The specific formula used for training the proposed structure-aware Transformer regression model is as follows:
[0058] in, and This represents the weights of the two losses.
[0059] The non-rigid registration model based on structure awareness proposed in this application solves the problems of cross-modal and cross-dimensional issues, insufficient image quality, and tissue deformation compared to existing supervised intraoperative two-dimensional ultrasound image to preoperative three-dimensional image registration methods.
[0060] The unsupervised training strategy proposed in this application, which combines real and synthetic images, addresses the significant shortcomings of existing unsupervised methods. On one hand, compared to supervised training methods that generate virtual synthetic images, the proposed strategy allows the model to perceive the structural features of real images, thereby improving the accuracy of real ultrasound image registration. On the other hand, compared to existing self-supervised learning methods that rely on image similarity between the input image and the synthetic image generated from the registration result, the proposed method enables the model to perceive the accurate optimization direction in three-dimensional space, thus effectively performing self-supervised learning and obtaining accurate registration results.
[0061] In summary, this application proposes an unsupervised visual navigation architecture guided by structure contour maps. Through a structure-aware Transformer regression model and a joint unsupervised learning strategy for synthesizing real images, it effectively performs non-rigid registration of intraoperative ultrasound to preoperative images. This addresses the significant lack of registration accuracy in existing unsupervised ultrasound visual navigation methods, achieving more efficient and accurate intraoperative ultrasound to preoperative image registration.
[0062] This application also proposes a surgical robot navigation system, such as Figure 3As shown, the system mainly consists of a multi-degree-of-freedom robotic arm, an ultrasound imaging probe, and a control platform (including a high-performance computer and peripherals such as a monitor). It utilizes existing hand-eye calibration algorithms to achieve spatial coordinate transformation between the robotic arm and the probe. Through a digital video image adapter, the ultrasound probe is connected to the control platform to display real-time ultrasound images. Simultaneously, the control platform is equipped with a self-developed software system for visualizing intraoperative multimodal data (such as preoperative medical images and their 3D reconstruction models).
[0063] For the AI-driven reconstruction of preoperative and intraoperative ultrasound images: preoperative images such as CT / MR images are segmented and reconstructed to obtain a three-dimensional digital vascular model; intraoperative ultrasound images are segmented and reconstructed to obtain an ultrasound digital vascular model.
[0064] Finally, there is the unsupervised (without physician operation) surgical robot autonomous navigation operation of the ultrasound probe: the autonomous navigation technology based on the artificial intelligence structure perception Transformer regression model and the artificial intelligence combined with the unsupervised learning strategy of real synthetic images, which is the multi-modal medical image registration method of this application, enables the robotic arm to autonomously control the ultrasound probe to realize the disease examination and surgical treatment process.
[0065] Based on the same inventive concept as the above disclosure, this disclosure also provides an electronic device. The electronic device of this disclosure includes at least one processor and at least one memory electrically connected to the processor. The memory is electrically connected to the processor, wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method described above.
[0066] It should be noted that the electrical connection between the above-mentioned units does not necessarily mean the connection between lines. The indirect connection method can be applied to the embodiments of this disclosure as long as it achieves the purpose of this disclosure.
[0067] Based on the same inventive concept as the above-disclosed content, this disclosure also provides a surgical robot navigation system, including the aforementioned electronic device.
[0068] Based on the same inventive concept, this disclosure also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the above-described method. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to depart from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A multimodal medical image registration method, characterized in that, include: Pre-processing was performed on preoperative 3D images and intraoperative 2D ultrasound images to obtain 3D structural data and 2D structural image sequences. The three-dimensional structure data and the two-dimensional structure image sequence are input into a pre-trained structure-aware Transformer regression model; The input data is processed using a trained structure-aware Transformer regression model to output a non-rigid registration coordinate map, thereby achieving spatial registration from intraoperative two-dimensional ultrasound images to preoperative three-dimensional images. The structure-aware Transformer regression model includes a location-aware image patch encoding module and a feature extraction and regression prediction module. The position-aware image block encoding module is used to encode and remove redundant information from the input three-dimensional structure data and two-dimensional structure image sequence based on spatial and temporal position priors, and to map the processed image blocks into three-dimensional data tokens and two-dimensional data tokens. The feature extraction and regression prediction module is used to perform self-attention feature extraction on the three-dimensional data token and the two-dimensional data token respectively to obtain three-dimensional global features and two-dimensional local features; the three-dimensional global features are fused into the two-dimensional local features through a one-way cross-modal attention mechanism to obtain enhanced two-dimensional features; and spatial coordinate regression is performed based on the enhanced two-dimensional features to generate the non-rigid registration coordinate map.
2. The method according to claim 1, characterized in that, The position-aware image block encoding module is specifically used to generate a three-dimensional position map for marking the spatial position of voxels for the three-dimensional structure data, and to generate a two-dimensional position map for marking pixel coordinates and time sequence for each frame image in the two-dimensional structure image sequence. The 3D location map and the 3D structure data are concatenated by channels to obtain 3D enhanced features. The 2D location map and the corresponding 2D structure image are concatenated by channels to obtain 2D enhanced sequences. The 3D augmented data and the 2D augmented sequence are respectively encoded into 3D image blocks and 2D image blocks according to preset sizes to obtain 3D image blocks and 2D image blocks; invalid image blocks in the 3D image blocks and all image blocks in the 2D augmented sequence except the last frame are removed, and all image blocks in the last 2D image are retained to obtain a set of 3D valid image blocks and a set of 2D valid image blocks; the set of 3D valid image blocks and the set of 2D valid image blocks are encoded by a multilayer perceptron to generate a 3D data token containing global spatial structure information and a 2D data token containing local temporal features, respectively.
3. The method according to claim 1, characterized in that, The feature extraction and regression prediction module is specifically used to perform self-attention feature extraction on the three-dimensional data token and the two-dimensional data token using L1 and L2 Transformer layers respectively to obtain three-dimensional global features and two-dimensional local features. Then, L3 Transformer layers are used to perform unidirectional cross-attention extraction, injecting the three-dimensional global features into the two-dimensional local features to perform cross-modal feature fusion and obtain enhanced two-dimensional features. Extract the target data token corresponding to the last frame of the two-dimensional image from the enhanced two-dimensional features; The target feature tokens are sequentially subjected to spatial dimension reshaping, upsampling, and convolutional mapping to generate a non-rigid registration coordinate map, where L1, L2, and L3 are positive integers greater than 1.
4. The method according to claim 1, characterized in that, The training process of the structure-aware Transformer regression model is as follows: Structural segmentation is performed on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively. Based on the three-dimensional structure data, simulated scanning generates a sequence of synthesized two-dimensional structure images, and the synthesized ground truth coordinate map is recorded; Based on the three-dimensional structure data, a three-dimensional contour map with gradient decay characteristics is constructed according to the distance from the voxel to the nearest structural edge. The three-dimensional structure data and the synthesized two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a synthesized predicted coordinate map; the coordinate regression constraint loss is calculated based on the synthesized predicted coordinate map and the synthesized ground truth coordinate map. The three-dimensional structure data and the two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a true predicted coordinate map. Based on the true predicted coordinate map and the three-dimensional contour map, the contour line guided constraint loss is calculated. By combining the coordinate regression constraint loss and the contour line guided constraint loss, the structure-aware Transformer regression model is trained end-to-end without supervision.
5. The method according to claim 1, characterized in that, The training process of the structure-aware Transformer regression model is as follows: Structural segmentation is performed on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively. Based on the three-dimensional structure data, simulated scanning generates a sequence of synthesized two-dimensional structure images, and the synthesized ground truth coordinate map is recorded; The three-dimensional structure data and the synthesized two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a synthesized predicted coordinate map; the coordinate regression constraint loss is calculated based on the synthesized predicted coordinate map and the synthesized ground truth coordinate map. The three-dimensional structure data and the two-dimensional structure image sequence are input into the structure-aware Transformer regression model, which outputs a true predicted coordinate map and generates a synthetic image based on the true predicted coordinate map. Calculate the effective region constraint loss based on two-dimensional structured images, synthetic images, and effective region masks of two-dimensional structured images; By combining the coordinate regression constraint loss and the effective region constraint loss, the structure-aware Transformer regression model is trained end-to-end without supervision.
6. The method according to claim 1, characterized in that, Preoperative 3D images and intraoperative 2D ultrasound images were preprocessed to obtain 3D structural data and 2D structural image sequences, including: The system performs structural segmentation on the input preoperative 3D images and 2D structural image sequences to obtain 3D structural data and 2D structural image sequences, respectively.
7. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes execution instructions stored in the memory, causing the processor to perform the method of any one of claims 1 to 6.
8. A surgical robot navigation system, characterized in that, include: The electronic device according to claim 7.
9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the method of any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.