A multi-modal feature fusion-based real-time image registration method for surgical robots

Abstract

The application discloses a surgical robot real-time image registration method based on multi-modal feature fusion. The method comprises the following steps: acquiring preoperative and intraoperative medical images of different modalities; extracting depth features by using first and second lightweight feature extractors, wherein the second extractor is dynamically adjusted according to real-time quality evaluation of the intraoperative images; mapping the depth features to the same latent space to obtain latent space representation by using a semantic feature conversion module comprising a pre-trained segmentation model and a projection network trained by contrast loss; obtaining fusion features by using an uncertainty-modulated cross-modal attention mechanism to fuse the latent space representation; and realizing image real-time registration by regressing a dense displacement field based on the fusion features through a deep learning network. The application significantly improves the accuracy and robustness of multi-modal image registration in a complex surgical environment, meets the real-time navigation requirements, and provides reliable guidance for surgeons.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image data processing, in particular to a surgical robot real-time image registration method based on multi-modal feature fusion. BACKGROUND

[0002] The emergence and development of image-guided surgery (IGS) have significantly improved the precision and safety of surgery, and the image registration technology plays a crucial role. By accurately aligning medical images of different times (such as preoperative and intraoperative) or different modalities (such as CT, MRI, ultrasound, and endoscopy), surgeons can obtain more comprehensive lesion information, clearer anatomical structures, and more reliable surgical navigation, which is of great significance for achieving minimally invasive surgery, reducing complications, and improving patient outcomes. The commonly used imaging modalities in clinical practice have their own advantages, such as CT providing high-resolution anatomical structures, MRI having excellent contrast for soft tissues, and ultrasound (US) and endoscopy providing real-time feedback. Multi-modal image registration can overcome the limitations of a single modality by fusing these complementary information.

[0003] However, existing multi-modal surgical image registration techniques still face many challenges, especially in the application scenarios of surgical robots requiring real-time feedback. First, the inherent geometric distortion between different modalities, image attribute differences (such as intensity, texture, resolution), and noise and artifacts seriously affect the accuracy and robustness of registration. In particular, in soft tissue surgery, organ deformation and brain tissue drift caused by breathing and instrument operation make accurate alignment particularly difficult. Second, the real-time requirement of the registration algorithm is extremely high during surgery, and traditional optimization-based methods often consume a lot of time, making it difficult to meet clinical needs. Although deep learning methods have improved speed, complex models may still face computational bottlenecks, and their "black box" characteristics, dependence on large-scale high-quality labeled data sets, and insufficient generalization ability also limit their clinical application and the establishment of physician trust.

[0004] In addition, how to effectively fuse heterogeneous features from different modalities, extract and align semantic information with clear anatomical significance, rather than just relying on apparent intensity or texture, is the key to improving registration quality. In existing technologies, some methods attempt to convert images into modality-neutral representations, but may not perform well in areas with subtle texture differences but clear semantic significance, and do not fully consider the adaptive impact of image quality or uncertainty on the fusion process, which is also a major obstacle to their widespread clinical application. Therefore, there is an urgent need for a multi-modal image registration scheme that can respond to various complex intraoperative situations in real time, accurately, and robustly, and perform intelligent and interpretable feature fusion. SUMMARY

[0005] The application provides a surgical robot real-time image registration method based on multi-modal feature fusion, which is used for solving the problems that in the complex situation of in-situ image quality change, noise artifact interference, soft tissue deformation and the like, it is difficult to balance high registration accuracy, strong robustness and high real-time performance, and in particular, the existing method has the deficiency in effectively fusing deep semantic information of different modal images to guide registration.

[0006] According to an aspect of the application, a surgical robot real-time image registration method based on multi-modal feature fusion is provided, the method comprises:

[0007] obtaining preoperative medical images of a first mode and in-situ medical images of a second mode;

[0008] extracting first deep features from the obtained preoperative medical images by using a first lightweight feature extractor, and extracting second deep features from the obtained in-situ medical images by using a second lightweight feature extractor, wherein the second lightweight feature extractor is dynamically self-adaptively adjusted according to a real-time image quality evaluation result of the in-situ medical images;

[0009] converting the first deep features and the second deep features to the same target latent space by using a semantic feature conversion module to obtain first latent space representation and second latent space representation respectively, the semantic feature conversion module comprises a pre-trained segmentation model and a learning projection network trained by a contrast loss function, the segmentation model is used for extracting high-level anatomical features based on the first deep features and the second deep features, and the learning projection network is used for mapping the high-level anatomical features of different modes to the target latent space;

[0010] fusing the first latent space representation and the second latent space representation by using a cross-modal attention mechanism to obtain fused features;

[0011] estimating registration parameters for aligning the preoperative medical images and the in-situ medical images based on the fused features, so as to register the preoperative medical images and the in-situ medical images in real time.

[0012] In some embodiments, the preoperative medical images of the first mode are computed tomography (CT) images or magnetic resonance imaging (MRI) images, and the in-situ medical images of the second mode are three-dimensional ultrasound (3D US) images or stereoscopic endoscopic video images.

[0013] In some embodiments, the first lightweight feature extractor is a pre-trained three-dimensional convolutional neural network.

[0014] In some implementations, when the intraoperative medical image of the second modality is a three-dimensional ultrasound image, the backbone network of the second lightweight feature extractor adopts a hybrid network architecture including a shallow three-dimensional convolutional neural network and a Transformer encoder layer, wherein the Transformer encoder layer is applied to the block embedding of the feature map of the shallow three-dimensional convolutional neural network.

[0015] In some implementations, when the intraoperative medical image of the second modality is a stereoscopic endoscopic video image, the backbone network of the second lightweight feature extractor adopts a compact visual Transformer (ViT) architecture with reduced number of layers and attention heads.

[0016] In some implementations, the image quality assessment result is obtained through a lightweight convolutional neural network branch that assists the second lightweight feature extractor. The input of the lightweight convolutional neural network branch is the intraoperative medical image, and the output is the image quality assessment result, which includes some or all of the signal-to-noise ratio, contrast, and artifact level.

[0017] In some implementations, the second lightweight feature extractor dynamically and adaptively adjusts based on real-time image quality assessment results of intraoperative medical images, including:

[0018] Based on the image quality assessment results, the filter parameters of the second lightweight feature extractor are adjusted through a gating mechanism, or the corresponding network layer or attention head of the second lightweight feature extractor is selectively activated or deactivated.

[0019] In some implementations, the pre-trained segmentation model is a TotalSegmentator model, and the feature map after each downsampling stage in its encoder path is used as the high-level anatomical feature.

[0020] In some implementations, the learning projection network is a shallow multilayer perceptron (MLP) or a small autoencoder structure, and the contrastive loss function is the InfoNCE loss defined according to the following formula:

[0021] ,

[0022] in, It is from the first mode. The representation of a sample in the target latent space, It is the representation of the corresponding positive sample from the second modality in the target latent space. and Representing negative samples, It refers to the batch size. Indicates calculation and cosine similarity, It is a temperature hyperparameter.

[0023] In some implementations, the cross-modal attention mechanism further includes the following uncertainty modulation:

[0024] Before fusion, the feature uncertainty associated with the second latent space representation is estimated, and the contribution weights of the first and second latent space representations in the fusion process are adjusted based on the estimated feature uncertainty.

[0025] In some implementations, the feature uncertainty includes random uncertainty and cognitive uncertainty, wherein the random uncertainty is obtained by modifying the last layer of the second lightweight feature extractor to predict the probability distribution parameters of the second deep feature, and the cognitive uncertainty is estimated by applying the Monte Carlo Dropout method to the second lightweight feature extractor or by employing a deep ensemble method.

[0026] In some implementations, the uncertainty modulation is achieved in the following ways:

[0027] The first latent space representation of the first latent space 1 feature vector as query The second latent space is represented by the first... The eigenvectors, after linear transformation, serve as the keys. Calculate query s and keys Original similarity scores between ;

[0028] Original similarity score With weight Multiply by each other to obtain the adjusted similarity score. , , For hyperparameters greater than zero, The second latent space representation of the first The feature uncertainty corresponding to each feature vector;

[0029] Use the adjusted similarity score Calculate attention weights, which are used to transform the feature vectors in the second latent space representation into a value vector by another linear transformation. Perform a weighted sum to obtain the result for the query. Context-related feature vectors;

[0030] Will target the query Context-related feature vectors and queries By combining preset fusion operations, fused features are generated.

[0031] In some implementations, the uncertainty modulation is achieved in the following ways:

[0032] The first latent space representation of the first latent space 1 feature vector as query The second latent space is represented by the first... The eigenvectors, after linear transformation, serve as the keys. Calculate query s and keys Attention weights between them;

[0033] According to the second latent space representation, the first The feature uncertainty corresponding to each feature vector For the second latent space representation of the first The value vector obtained by another linear transformation of the eigenvectors Scale the vector to obtain the adjusted value vector. ;

[0034] In the attention mechanism, the calculated attention weights are used to adjust the value vector. Perform a weighted sum to obtain the result for the query. Context-related feature vectors;

[0035] Will target the query Context-related feature vectors and queries By combining preset fusion operations, fused features are generated.

[0036] In some implementations, registration parameters for aligning the preoperative medical images and the intraoperative medical images are estimated based on the fused features, including:

[0037] The fused features are input into a deep neural network with a VoxelMorph-like U-Net architecture, and the deep neural network is used to regress the dense displacement field as the registration parameter. The deep neural network is optimized using a composite loss function, which includes semantic similarity loss and regularization loss. The semantic similarity loss is obtained by calculating the similarity between the features obtained by deforming the first latent space representation according to the estimated dense displacement field in the target latent space and the second latent space representation. The regularization loss is obtained by calculating the norm of the gradient of the estimated dense displacement field or its bending energy term.

[0038] The proposed real-time image registration method for surgical robots based on multimodal feature fusion significantly improves the accuracy and robustness of multimodal image registration in complex surgical environments through dynamic adaptive feature extraction, transformation to a modally neutral latent space rich in anatomical meaning, feature fusion with uncertainty modulation, and semantic-based registration optimization. This method not only meets the stringent real-time requirements of surgical navigation but also provides surgeons with more reliable intraoperative guidance through deeper semantic understanding and adaptation to changes in data quality, while reducing reliance on external markers. Attached Figure Description

[0039] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0040] Figure 1 A flowchart of a real-time image registration method for a surgical robot based on multimodal feature fusion according to an embodiment of this application is shown.

[0041] Figure 2 A block diagram of a real-time image registration system for a surgical robot based on multimodal feature fusion according to an exemplary embodiment of this application is shown. Detailed Implementation

[0042] To enable those skilled in the art to better understand and implement the technical solutions of this application, the real-time image registration method for surgical robots based on multimodal feature fusion proposed in this application will be described in detail below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0043] Reference Figure 1 This application provides a real-time image registration method for surgical robots based on multimodal feature fusion, which includes the following steps 101 to 105.

[0044] Step 101: Acquire preoperative medical images of the first modality and intraoperative medical images of the second modality.

[0045] According to this embodiment, medical image data from two different sources are first acquired for registration.

[0046] The first modality of preoperative medical images typically consists of high-quality, high-resolution image data acquired before surgery, used for surgical planning and as a reference for registration. In some implementations, the first modality of preoperative medical images can be computed tomography (CT) images or magnetic resonance imaging (MRI) images. For example, preoperative CT images (e.g., 512x512xZ slices, 1mm isotropic resolution) or T1-weighted MRI images containing the target lesion area can be acquired.

[0047] The second modality of intraoperative medical imaging consists of image data acquired in real-time or near real-time during the surgical procedure, used to reflect current anatomical structures and surgical progress. In some embodiments, the second modality of intraoperative medical imaging may be three-dimensional ultrasound (3D US) images or stereoscopic endoscopic video images. For example, during surgery, volumetric data (e.g., 200x150x100 voxels, 5-10Hz) can be acquired in real time via a tracked 3D ultrasound transducer, or continuous image pairs (e.g., 1920x1080 resolution, 30Hz) can be acquired via a stereoscopic endoscope.

[0048] After acquiring the images of the first and second modalities, data synchronization and preprocessing can be performed as needed. In particular, for the intraoperative medical images (e.g., 3D ultrasound or stereoscopic endoscopic video) of the second modality acquired in this application, temporal and spatial synchronization with the surgical robot system or external tracking system (e.g., optical tracking system) can be achieved or assisted in being achieved through the following methods.

[0049] Precise timestamps can be applied to the real-time intraoperative image data stream and the status data stream of the surgical robot / tracking system (such as robot joint angles, end effector positions, or spatial positioning information of the ultrasound probe / endoscope) to achieve time synchronization. Where feasible, hardware triggering mechanisms (e.g., triggering an image acquisition after the tracking system completes positioning) or software methods such as High Precision Network Time Protocol (NTP) can be used to ensure a strict temporal correspondence between these data from different sources.

[0050] Spatial calibration can be performed on intraoperative imaging equipment (such as ultrasound probes or endoscopes) to determine the relative positional relationship between its imaging coordinate system and physical space or tracking markers. The surgical robot can also be calibrated to ensure that the images are aligned with surgical instruments or anatomical structures in the same physical space.

[0051] Furthermore, according to some embodiments of this application, bilateral filters can be used to despeccize intraoperative three-dimensional ultrasound images to improve image quality; gamma correction or dehazing can be performed on stereoscopic endoscopic video images to enhance image clarity. Intensity normalization (e.g., Z-score normalization) and image resampling (e.g., unification to isotropic voxel spacing) can be performed on all image data (including preoperative images and preprocessed intraoperative images), which helps to improve the stability and accuracy of subsequent feature extraction and registration.

[0052] Step 102: Extract a first depth feature from the acquired preoperative medical images using a first lightweight feature extractor, and extract a second depth feature from the acquired intraoperative medical images using a second lightweight feature extractor, wherein the second lightweight feature extractor is dynamically and adaptively adjusted based on the real-time image quality assessment results of the intraoperative medical images.

[0053] In this step, different lightweight feature extractors can be used to extract depth features for images of different modalities, and a dynamic adaptive mechanism is introduced specifically for feature extraction of intraoperative images.

[0054] In some implementations, the first lightweight feature extractor can be a pre-trained 3D convolutional neural network (CNN). For example, a 3D variant of a compact ResNet or EfficientNet pre-trained on a large medical image dataset can be used to extract multi-scale first depth features from preoperative CT or MRI images.

[0055] In some implementations, in order to better adapt to the real-time nature and quality variability of intraoperative images, a second lightweight feature extractor can be designed specifically for different modalities of intraoperative medical images.

[0056] When the intraoperative medical image of the second modality is a three-dimensional ultrasound image, the backbone network of the second lightweight feature extractor can adopt a hybrid network architecture including a shallow three-dimensional convolutional neural network and a Transformer encoder layer (e.g., a hybrid CNN-Transformer architecture), wherein the Transformer encoder layer is applied to block embedding of the feature map of the shallow three-dimensional convolutional neural network. The second lightweight feature extractor based on this design can efficiently extract local texture and structural features of ultrasound images and better overcome noise and artifact problems.

[0057] In some examples, this hybrid CNN-Transformer architecture can begin with a shallow 3D CNN (e.g., containing 3-4 convolutional modules). This CNN module is responsible for efficiently extracting local 3D texture and basic structural features from the input raw 3D ultrasound body data, such as echo patterns of tissues, cross-sections of small blood vessels, etc. Its design can draw on lightweight CNN concepts, such as using smaller convolutional kernels (e.g., 3x3x3), straddle convolutions, or depthwise separable convolutions to optimize parameters and computation. To capture longer-range spatial dependencies and global contextual information, which is crucial for accurately understanding the overall contours and relative positions of organs in noisy, blurred-boundary ultrasound images, this hybrid network architecture can further apply one or more small Transformer encoder layers to the feature maps output from the CNN module. For example, the 3D feature maps output by the CNN can be segmented into patches, these patches can be linearly embedded into sequence data with positional encoding, and then this sequence can be fed into the Transformer encoder. The Transformer's self-attention mechanism can compute the interdependencies between different patches in the sequence, thereby capturing the global context. Furthermore, this hybrid network architecture fully considers the requirements for lightweight characteristics in order to better meet the real-time requirements during surgery.

[0058] When the intraoperative medical image of the second modality is a stereoscopic endoscopic video image, the backbone network of the second lightweight feature extractor preferably adopts a compact Visual Transformer (ViT) architecture with reduced layer and attention head count. The second lightweight feature extractor designed according to this principle can efficiently process the features of endoscopic video frames while meeting the real-time requirements of the surgical scenario.

[0059] In some examples, this compact Vision Transformer (ViT) architecture can use input processing and patch embedding. For each frame of the input stereoscopic endoscopic video (or a frame in the left or right view), it is first segmented into a series of fixed-size, non-overlapping or partially overlapping 2D image patches. These 2D image patches are then linearly flattened and embedded into a lower-dimensional vector space through a learnable linear projection layer, forming a patch embedding sequence. To preserve the spatial positional information of the patches, learnable positional encodings are typically added to the patch embedding.

[0060] The sequence, after patch embedding and positional encoding, is then input into a compact Transformer encoder. This encoder can consist of multiple stacked Transformer layers. To achieve "compactness," compared to the standard ViT model, the number of layers in the Transformer encoder used here (e.g., fewer Transformer layers) and the number of attention heads in each Multi-Head Self-Attention (MHSA) module can be reduced, significantly reducing the model's parameter count and computational complexity, thus meeting the demands of real-time processing. The MHSA module in each Transformer layer allows the model to learn dependencies between different regions (patches) of the image in parallel across different representation subspaces, thereby capturing global contextual information and important features of the image. Even with the reduced number of attention heads, the compact model according to this embodiment can still effectively extract key information.

[0061] Each Transformer layer typically also contains one or more fully connected feedforward network layers for nonlinear transformation of the output of the self-attention mechanism. The final output of the Transformer encoder (e.g., a sequence of all patch feature representations, or an aggregated global feature representation) is post-processed appropriately, such as dimensionality adjustment or format uniformization, to ensure a structure compatible with features extracted in the context of 3D ultrasound, and constitutes the second depth feature.

[0062] This compact ViT architecture, optimized for endoscopic video, can effectively capture global and local visual features within image frames while maintaining low computational overhead. Furthermore, the features it extracts are of great significance for understanding information such as scenes, organ surfaces, and surgical instruments under endoscopy.

[0063] The second lightweight feature extractor according to this application integrates a dynamic adaptive adjustment mechanism, which can dynamically adjust the behavior of the backbone network based on real-time image quality assessment results, thereby extracting more robust and effective depth features under various imaging conditions to cope with the real-time changes in image quality of intraoperative medical images.

[0064] In some implementations, the image quality assessment result is obtained through an auxiliary lightweight convolutional neural network (CNN) branch in the second lightweight feature extractor.

[0065] This auxiliary CNN branch can directly receive raw intraoperative medical images (e.g., current 3D ultrasound body data frames or stereoscopic endoscopy video frames, or their rapidly downsampled versions) as input, and its output is a real-time image quality assessment result, which is typically a vector, with each component of the vector corresponding to a quantized value of a different image quality index.

[0066] In some examples, image quality assessment results may include some or all of the following: signal-to-noise ratio (SNR), image contrast, and the degree of artifacts (such as acoustic shadowing in ultrasound images, smoke or specular highlights in endoscopic images).

[0067] In some examples, the auxiliary CNN branch itself can be designed to be very lightweight, for example, consisting of a few convolutional layers (such as 3x3x3 convolutional kernels with stride controlling the downsampling speed), activation functions (such as ReLU), a global average pooling layer, and one or more fully connected layers, ultimately outputting the aforementioned quality evaluation vector to quickly and accurately assess the overall image quality of the current frame.

[0068] In some implementations, the backbone network of the second lightweight feature extractor (e.g., a hybrid CNN-Transformer architecture, a compact ViT architecture, etc.) can be dynamically and adaptively adjusted based on the real-time image quality assessment result (e.g., output by the aforementioned auxiliary CNN branch). For example, the filter parameters of the second lightweight feature extractor can be adjusted through a gating mechanism, or the corresponding network layer or attention head of the second lightweight feature extractor can be selectively activated or deactivated.

[0069] For CNN-based components (e.g., shallow 3D CNN modules in a hybrid CNN-Transformer architecture), the filter weights of convolutional layers can be dynamically adjusted or different preset filter banks can be selected based on image quality assessment results (e.g., low signal-to-noise ratio or high artifacts). This can be achieved through a gating unit that receives real-time image quality assessment results and outputs control signals to modulate the filter behavior.

[0070] For networks containing Transformer layers (such as the Transformer encoder module in a hybrid CNN-Transformer architecture, or a compact ViT architecture), certain Transformer layers or their internal attention heads can be dynamically activated or "pruned" based on real-time image quality assessment results. For example, when image quality is very poor, the network can be designed to dynamically reduce the number of active Transformer layers or their attention heads. Alternatively, in some cases, if the overall network structure allows, a gating mechanism can be used to control the computational path, prioritizing shallower or less computationally intensive layers. The gating signal is generated by passing the real-time image quality assessment results through a small gating unit (e.g., one or more fully connected layers followed by a Sigmoid activation function). The gating value (between 0 and 1) can be directly multiplied by the output of the corresponding attention head or used to scale the output of the entire network layer.

[0071] These adjustments enable the second lightweight feature extractor to fully utilize its capabilities to extract fine features when image quality is good, while reducing its sensitivity to noise and artifacts when image quality deteriorates, thus extracting more robust macroscopic structural information and improving the overall quality and stability of the second deep feature.

[0072] Step 103: The semantic feature conversion module is used to convert the first deep feature and the second deep feature to the same target latent space, thereby obtaining the first latent space representation and the second latent space representation, respectively. The semantic feature conversion module includes a pre-trained segmentation model and a learned projection network trained by a contrastive loss function. The segmentation model is used to extract high-level anatomical features based on the first deep feature and the second deep feature, and the learned projection network is used to map high-level anatomical features of different modalities to the target latent space.

[0073] The purpose of this step is to map deep features from different modalities and with different characteristics into a common, more abstract, and anatomically rich latent space to facilitate subsequent comparison and fusion.

[0074] First, both the first and second deep features extracted in step 102 undergo semantic abstraction. A pre-trained segmentation model can be used here; in some embodiments, this pre-trained segmentation model can be a TotalSegmentator model, using the feature maps after each downsampling stage in its encoder path as the high-level anatomical features. According to this embodiment, high-level information with clear anatomical significance can be further extracted from the deep features, such as distinguishing different tissue types and organ boundaries, rather than just low-level textures or edges.

[0075] Then, a learned projection network can be used to map these high-level anatomical (semantic) features from different modalities to the same target latent space. This learned projection network can be a shallow multilayer perceptron (MLP) or a small autoencoder structure. To ensure that corresponding anatomical structures from different modalities map to similar representations in this latent space, while different structures map to more distant points, the projection network can be trained using a contrastive loss function. In some implementations, the InfoNCE loss, defined as follows:

[0076] ,

[0077] in, It is from the first mode. The representation of a sample in the target latent space, It is the representation of the corresponding positive sample from the second modality in the target latent space. and Representing negative samples, It refers to the batch size. Indicates calculation and cosine similarity, It is a temperature hyperparameter, which is usually a preset and adjustable parameter.

[0078] Through the above process, the original modality-specific deep features are transformed into modality-invariant (or modality-neutral) latent space representations rich in anatomical semantics, namely the first latent space representation and the second latent space representation.

[0079] Step 104: Using a cross-modal attention mechanism, the first latent space representation and the second latent space representation are fused to obtain the fused features.

[0080] In this step, feature representations from different modalities that are already in the same target latent space are effectively fused to combine their respective advantages.

[0081] According to embodiments of this application, a cross-modal attention mechanism is employed for feature fusion. The cross-modal attention mechanism allows features from one modality (e.g., the latent space representation of preoperative CT) to focus on features from another modality (e.g., the latent space representation of intraoperative ultrasound) and vice versa, thereby learning the correlation between them and identifying complementary information.

[0082] In some implementations, the cross-modal attention mechanism further includes uncertainty modulation, specifically, prior to fusion, feature uncertainty related to the second latent space representation (i.e., features derived from intraoperative images) can be estimated, and the contribution weights of the first and second latent space representations in the fusion process can be dynamically adjusted based on the estimated uncertainty, such that the contribution weight of the latent space representation with higher uncertainty in the fusion process is reduced.

[0083] In some implementations, the characteristic uncertainty may include both accidental uncertainty and cognitive uncertainty.

[0084] Random uncertainty typically reflects inherent noise, ambiguity, or incomplete information in the data itself. For example, in intraoperative ultrasound imaging, due to limitations in imaging principles, speckle noise, acoustic shadowing, or unclear tissue boundaries may occur, all of which can lead to random uncertainty in the extracted second depth features.

[0085] In some implementations, this can be achieved by modifying the network structure of the second lightweight feature extractor, particularly its last layer (i.e., the output layer), so that it predicts not only the second deep features themselves but also the probability distribution parameters of these features. For example, assuming that each element of the second deep feature follows an independent Gaussian distribution, the output layer of the second lightweight feature extractor can be designed to predict two values ​​simultaneously for each feature dimension: the mean of the Gaussian distribution of that dimension's feature (…). ) and variance ( The predicted mean (μ) can be considered the best estimate of the feature, i.e., the second-depth feature itself. The predicted variance (σ²) directly quantifies the uncertainty of the feature element due to factors such as inherent noise in the data. The larger the variance, the higher the uncertainty of the feature element.

[0086] If the second deep feature is a feature map (e.g., the output of a CNN), then a mean and variance can be predicted for each channel of each pixel / voxel on the feature map, forming an uncertainty map of the same size as the feature map. To teach the network to predict meaningful variance, it can be trained using a specific loss function, such as Negative Log-Likelihood (NLL). This loss function penalizes the difference between the predicted mean and the target value, and encourages the model to predict larger variances where the data points are inherently fuzzy or noisy. In this way, the second lightweight feature extractor can output the second deep feature and its accompanying pixel / voxel-level random uncertainty map.

[0087] Cognitive uncertainty reflects the uncertainty inherent in a model due to insufficient training data, imperfect model structure, or incomplete parameter convergence. It represents the model's confidence in its predictions. Cognitive uncertainty is typically high when the model encounters unseen or significantly different data in the training set. In some implementations, cognitive uncertainty can be estimated by applying Monte Carlo Dropout to a second lightweight feature extractor or by employing deep ensemble methods.

[0088] Monte Carlo Dropout (MC Dropout) is a regularization technique used for training neural networks that prevents overfitting by randomly deactivating a subset of neurons during each forward propagation. The core idea of ​​MC Dropout is to maintain Dropout activation during the inference (test) phase. For example, for the same intraoperative medical image input, multiple (e.g., typically dozens to hundreds) forward propagations can be performed using a second lightweight feature extractor (whose internal Dropout layer remains active during inference). Because different neurons are randomly deactivated during each propagation, multiple slightly different second-depth feature outputs are obtained. The statistical properties of these outputs (e.g., calculating their variance or standard deviation across each feature dimension) can serve as a measure of cognitive uncertainty. Similarly, if the output is a feature map, a cognitive uncertainty map can be obtained.

[0089] Deep ensembles are achieved by training multiple (e.g., M) second lightweight feature extractor models with identical structures but different initializations (or trained using different data subsets or in different training orders). For example, for the same intraoperative medical image input, inference can be performed through these M independent models, resulting in M ​​different second deep feature outputs. The consistency or dissimilarity among these M model outputs (e.g., calculating their variance in each feature dimension) can be used to quantify cognitive uncertainty. The greater the dissimilarity of the model outputs, the higher the cognitive uncertainty.

[0090] In practical applications, those skilled in the art can combine the estimated random uncertainty map and the cognitive uncertainty map as needed (e.g., by simple addition or more complex combinations) to obtain a comprehensive feature uncertainty.

[0091] In some implementations, the uncertainty modulation can be achieved by: representing the first latent space in the first latent space... 1 feature vector as query The second latent space is represented by the first... The eigenvectors, after linear transformation, serve as the keys. Calculate query s and keys Original similarity scores between ; the original similarity score With weight Multiply, , For hyperparameters greater than zero, The second latent space representation of the first The feature uncertainty corresponding to each feature vector yields the adjusted similarity score. Use the adjusted similarity score The attention weights are calculated and then used to process the value vectors corresponding to the feature vectors represented in the second latent space. Perform a weighted sum to obtain the result for the query. Context-related feature vectors; will be used for queries Context-related feature vectors and queries By combining preset fusion operations, fused features are generated.

[0092] The core idea of ​​this implementation is to adjust the original similarity or correlation between different features based on the uncertainty of the features before calculating the final attention weight.

[0093] In cross-modal attention mechanisms, there exists a query (Q) sequence and a key (K) sequence. According to this implementation, the feature vectors in the first latent space representation (derived from preoperative images) can be defined as the query sequence. The key sequences are obtained by performing different linear transformations on the feature vectors in the second latent space representation (derived from intraoperative images). Sum sequence .

[0094] To integrate the effects of uncertainty, computational queries can be performed. ( (a query vector) and key ( The original similarity score between (a key vector in the vector) For example, it can be calculated using the dot product. And utilize the key Corresponding characteristic uncertainty Generate modulation weights , These are hyperparameters that are greater than zero. Modulation weights. Then compared with the original similarity score Multiply by each other to obtain the adjusted similarity score. Based on the adjusted similarity score, the final attention weights can be calculated using the Softmax function. .

[0095] After obtaining attention weights that take uncertainty into account Then, these weights can be used. For the corresponding value vector Perform a weighted summation to obtain the result for the current query. Context-related feature vectors Context-dependent feature vectors It can be viewed as reliability-weighted information extracted from the second mode for the first mode.

[0096] Finally, the context-related feature vector can be... Its corresponding original query vector By combining preset fusion operations, such as splicing the two together and processing them through a linear layer, or adding elements together, or by using more complex gating mechanisms, their fusion ratio can be dynamically controlled.

[0097] In other embodiments, the uncertainty modulation is achieved by: modifying the first latent space representation in the first latent space representation... 1 feature vector as query and the second latent space representation of the first The eigenvectors, after linear transformation, serve as the keys. Calculate the attention weights between them; based on the second latent space representation... The feature uncertainty corresponding to each feature vector For the second latent space representation of the first The original value vector obtained by another linear transformation of the eigenvectors Scale the vector to obtain the adjusted value vector. In the attention mechanism, attention weights are used to adjust the value vector. Perform a weighted sum to obtain the result for the query. Context-related feature vectors; will be used for queries Context-related feature vectors and queries By combining preset fusion operations, fused features are generated.

[0098] The core concept of this implementation is to directly adjust the amount of information in the attention mechanism based on the degree of uncertainty of intraoperative features (i.e., scale their value vector), thereby reducing the overall contribution of unreliable intraoperative features when fused with preoperative feature information in the future.

[0099] Similarly, according to this embodiment, the feature vectors in the first latent space representation (derived from preoperative images) can be defined as query sequences. Simultaneously, the feature vectors in the second latent space representation (derived from intraoperative images) are subjected to different linear transformations to obtain key sequences. Sum sequence .

[0100] Based on this, attention weight The calculation can follow the standard method, that is, by querying... ( (a query vector) and key ( The similarity between two key vectors (e.g., after dot product and Softmax normalization) is calculated, and this weight reflects the degree of attention the query of the first modality pays to each part of the second modality.

[0101] Unlike the previous implementation, the impact of uncertainty in this implementation primarily affects the value vector. Based on the estimated characteristic uncertainty... For the original value vector Scale the vector to obtain the adjusted value vector. For example, it can be achieved through Scaling is applied, where γ is the scaling factor. This is applied when the feature uncertainty... At higher levels, The value is close to 1. Approaching 0, thus significantly reducing The modulus length; conversely, when the uncertainty is low, It will approach its original value. .

[0102] When the attention mechanism calculates the final output, the previously calculated attention weights are applied to the value vector after uncertainty adjustment. Perform a weighted summation to obtain the result for the current query. Context-related feature vectors ,For example In this way, even if certain intraoperative features receive high attention weights, their actual contribution to the weighted summation will be limited due to the decay of the value vector itself if their inherent uncertainty is high.

[0103] Finally, the context-related feature vector can be... Its corresponding original query vector By combining preset fusion operations, such as splicing the two together and processing them through a linear layer, or adding elements together, or by using more complex gating mechanisms, their fusion ratio can be dynamically controlled.

[0104] Both modulation methods described above can reduce the impact of intraoperative features with high uncertainty on attention calculation, while also ensuring that the information from the two modalities is effectively and reliably combined to obtain more robust and reliable fused features.

[0105] Step 105: Based on the fusion feature estimation, registration parameters are used to align preoperative and intraoperative medical images, so as to register preoperative and intraoperative medical images in real time.

[0106] In this step, the high-quality fusion features obtained in the previous step are used to drive the estimation of registration parameters, ultimately achieving image alignment.

[0107] In some implementations, the fused features output from the previous step can first be fed into a specially designed deep neural network. This deep neural network preferably employs a VoxelMorph-like U-Net architecture, which has been proven effective in learning complex spatial transformation relationships in image registration. After training, the network's task is to regress a dense displacement field (DDF), which constitutes the registration parameters used to align the two sets of images.

[0108] To ensure that the deep neural network can accurately learn the correct deformations, its training process can be optimized using a composite loss function. This composite loss function can consist of two main parts: semantic similarity loss and regularization loss.

[0109] The semantic similarity loss is computed in the target latent space and measures the alignment between deformed preoperative image features and intraoperative image features. More specifically, it is obtained by calculating the similarity between the features obtained by deforming the first latent space representation (corresponding to the original preoperative image) according to the dense displacement field predicted by the current network and the second latent space representation (corresponding to the feature representation of the original intraoperative image). This similarity can be quantified by calculating the L2 distance or negative cosine similarity between them. By comparing in a semantically rich latent space, more meaningful alignment of anatomical structures can be facilitated.

[0110] Meanwhile, the regularization loss is used to impose a constraint on the smoothness or physical plausibility of the predicted dense displacement field, which can typically be achieved by calculating the norm of the spatial gradient of the dense displacement field or by introducing a bending energy term that simulates physical deformation (such as elastic deformation). The regularization loss helps prevent unrealistic, overly distorted deformations, ensuring the smoothness and anatomical plausibility of the registration results.

[0111] By minimizing this composite loss function, the deep neural network is driven to learn to generate a dense displacement field that can accurately align images and has good physical properties, thereby achieving high-precision real-time image registration.

[0112] In some implementations, the registration process can be performed in a coarse-to-fine hierarchical manner, for example, by first estimating the global rigid or affine transformation parameters, and then using features at different scales from the fusion module to perform a more refined estimation of non-rigid deformations. This helps improve the robustness and efficiency of the registration.

[0113] Ultimately, the obtained registration parameters (such as DDF) are applied to the preoperative medical images, transforming them into the coordinate system of the intraoperative medical images, thereby achieving real-time registration between the two.

[0114] In some implementations, after registration, the deformed preoperative image (or its segmentation model) can be overlaid on real-time intraoperative images (such as endoscopic video) or presented on an augmented reality (AR) head-mounted device, providing surgeons with enhanced visualization and navigation capabilities. The image deformation process is typically accelerated on the GPU using efficient interpolation algorithms (such as B-spline interpolation or trilinear interpolation) to meet real-time requirements.

[0115] This application provides a high-precision, robust, and real-time-compliant multimodal surgical image registration method by combining an end-to-end process of dynamic adaptive feature extraction, semantic feature to modality-neutral latent space conversion, uncertain modulation-based adaptive cross-modal attention fusion, and semantic loss-based registration parameter estimation. This method can more effectively utilize the complementary information of different modalities and adapt to complex intraoperative environments, providing reliable navigation support for surgical robots and demonstrating significant clinical application potential.

[0116] To more intuitively understand the system implementation of the real-time image registration method for surgical robots based on multimodal feature fusion proposed in this application, please refer to... Figure 2 The figure schematically illustrates a system architecture block diagram of an exemplary embodiment of this application.

[0117] like Figure 2 As shown, the system first acquires preoperative medical images (e.g., CT or MRI) in the first modality and intraoperative medical images (e.g., US or endoscopy) in the second modality.

[0118] The first modality of preoperative medical images is fed into a first lightweight feature extractor, which (as described above, can be a pre-trained three-dimensional convolutional neural network) extracts the first depth features.

[0119] The second modality of intraoperative medical images undergoes a more complex processing path. The intraoperative medical images are fed into the backbone network of a second lightweight feature extractor (as mentioned earlier, this could be a CNN-Transformer hybrid architecture for ultrasound or a compact ViT architecture for endoscopy) to extract second-depth features. Simultaneously, the second-modality intraoperative medical images are also input to a real-time image quality assessment module (an auxiliary CNN branch), which outputs image quality assessment results. These assessment results are fed back to the backbone network of the second lightweight feature extractor for dynamic adaptive adjustments, ensuring robust extraction of second-depth features across different image quality levels.

[0120] Subsequently, the first and second deep features extracted by the first and second lightweight feature extractors, respectively, are both fed into the semantic feature conversion module. Within this module, a pre-trained segmentation model first extracts more anatomically meaningful high-level anatomical features from the input modally diverse deep features. Then, these high-level anatomical features are processed by a learned projection network (which can be trained using a contrastive loss function) and mapped onto the same target latent space, forming the first and second latent space representations.

[0121] Next, the feature fusion stage begins. The second latent space representation (derived from intraoperative images) is fed into the uncertainty estimation module, which estimates and outputs the feature uncertainty associated with this latent space representation. Subsequently, the first latent space representation, the second latent space representation, and the estimated feature uncertainty are fed together as input into the cross-modal attention mechanism module. This attention mechanism utilizes the feature uncertainty to modulate the fusion process (as described above, by modulating attention scores or value vectors), effectively combining the two latent space representations and ultimately outputting the fused features.

[0122] The fused features are then fed into a registration parameter estimation network (e.g., a VoxelMorph-like U-Net architecture). Based on the input fused features, the network learns and regresses registration parameters (typically dense displacement field DDFs) for aligning preoperative and intraoperative images.

[0123] Finally, these estimated registration parameters are fed into the real-time registration module. This module uses these parameters to transform the preoperative medical images into the coordinate system of the intraoperative medical images, thereby outputting aligned image / deformation fields to achieve real-time registration between the two and support surgical navigation.

[0124] pass Figure 2 The system structure shown in this application enables a systematic and efficient implementation of the entire process from multimodal image input to real-time registration output. The various modules work together, and in particular, the fusion mechanism of dynamic adaptive feature extraction and uncertainty modulation ensures the accuracy and robustness of registration in complex surgical environments.

[0125] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0126] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the data processing device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0127] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0128] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0129] The above description is merely a preferred embodiment of one or more embodiments of this specification and is not intended to limit the scope of one or more embodiments of this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this specification should be included within the protection scope of one or more embodiments of this specification.

Claims

1. A real-time image registration method for surgical robots based on multimodal feature fusion, characterized in that, The method includes: Acquire preoperative medical images of the first modality and intraoperative medical images of the second modality; A first lightweight feature extractor is used to extract a first depth feature from the acquired preoperative medical images, and a second lightweight feature extractor is used to extract a second depth feature from the acquired intraoperative medical images. The second lightweight feature extractor is dynamically and adaptively adjusted based on the real-time image quality assessment results of the intraoperative medical images. The semantic feature conversion module is used to convert the first deep feature and the second deep feature to the same target latent space, thereby obtaining the first latent space representation and the second latent space representation, respectively. The semantic feature conversion module includes a pre-trained segmentation model and a learned projection network trained by a contrastive loss function. The segmentation model is used to extract high-level anatomical features based on the first deep feature and the second deep feature, and the learned projection network is used to map high-level anatomical features of different modalities to the target latent space. By utilizing a cross-modal attention mechanism, the first latent space representation and the second latent space representation are fused to obtain fused features. The cross-modal attention mechanism includes the following uncertainty modulation: estimating the feature uncertainty related to the second latent space representation before fusion, and adjusting the contribution weights of the first latent space representation and the second latent space representation in the fusion process based on the estimated feature uncertainty. Registration parameters based on fusion feature estimation are used to align preoperative and intraoperative medical images, enabling real-time registration of preoperative and intraoperative medical images.

2. The method according to claim 1, characterized in that, The preoperative medical images of the first modality are computed tomography (CT) images or magnetic resonance imaging (MRI) images, and the intraoperative medical images of the second modality are three-dimensional ultrasound (3D US) images or stereoscopic endoscopic video images.

3. The method according to claim 1, characterized in that, The first lightweight feature extractor is a pre-trained three-dimensional convolutional neural network.

4. The method according to claim 2, characterized in that, When the intraoperative medical image of the second modality is a three-dimensional ultrasound image, the backbone network of the second lightweight feature extractor adopts a hybrid network architecture including a shallow three-dimensional convolutional neural network and a Transformer encoder layer, wherein the Transformer encoder layer is applied to the block embedding of the feature map of the shallow three-dimensional convolutional neural network.

5. The method according to claim 2, characterized in that, When the intraoperative medical image of the second modality is a stereoscopic endoscopic video image, the backbone network of the second lightweight feature extractor adopts a compact visual Transformer architecture with reduced number of layers and attention heads.

6. The method according to claim 1, characterized in that, The image quality assessment result is obtained through a lightweight convolutional neural network branch that assists the second lightweight feature extractor. The input of the lightweight convolutional neural network branch is the intraoperative medical image, and the output is the image quality assessment result. The image quality assessment result includes some or all of the signal-to-noise ratio, contrast, and artifact level.

7. The method according to claim 1, characterized in that, The second lightweight feature extractor dynamically and adaptively adjusts based on the real-time image quality assessment results of intraoperative medical images, including: Based on the image quality assessment results, the filter parameters of the second lightweight feature extractor are adjusted through a gating mechanism, or the corresponding network layer or attention head of the second lightweight feature extractor is selectively activated or deactivated.

8. The method according to claim 1, characterized in that, The pre-trained segmentation model is the TotalSegmentator model, and the feature map after each downsampling stage in its encoder path is used as the high-level anatomical feature.

9. The method according to claim 1, characterized in that, The learning projection network is a shallow multilayer perceptron (MLP) or a small autoencoder structure, and the contrastive loss function is the InfoNCE loss defined according to the following formula: in, It is the representation of the i-th sample from the first mode in the target latent space. It is the representation of the corresponding positive sample from the second modality in the target latent space. and τ represents negative samples, N is the batch size, sim(u,v) represents calculating the cosine similarity between u and v, and τ is the temperature hyperparameter.

10. The method according to claim 1, characterized in that: The feature uncertainty includes random uncertainty and cognitive uncertainty. The random uncertainty is obtained by modifying the last layer of the second lightweight feature extractor to predict the probability distribution parameters of the second deep feature. The cognitive uncertainty is estimated by applying the Monte Carlo Dropout method to the second lightweight feature extractor or by using a deep ensemble method.

11. The method according to claim 1, characterized in that, The uncertainty modulation is achieved in the following way: The i-th eigenvector in the first latent space representation is used as the query Q. Ai The j-th eigenvector in the second latent space representation is linearly transformed and used as the key K. Bj Calculate query Q Ai and key K Bj The original similarity scores between them s ij ; The original similarity score s ij With weight w j Multiply by each other to obtain the adjusted similarity score s′. ij w j =exp(-βU Bj ), where β is a hyperparameter greater than zero, U Bj The feature uncertainty corresponds to the j-th eigenvector in the second latent space representation; Using the adjusted similarity score s′ ij Calculate attention weights, which are used to transform the feature vectors in the second latent space representation into a value vector V through another linear transformation. Bj Perform a weighted sum to obtain the result for query Q. Ai Context-related feature vectors; Will target query Q Ai Context-related feature vectors and query Q Ai By combining preset fusion operations, fused features are generated.

12. The method according to claim 1, characterized in that, The uncertainty modulation is achieved in the following way: The i-th eigenvector in the first latent space representation is used as the query Q. Ai The j-th eigenvector in the second latent space representation is linearly transformed and used as the key K. Bj Calculate query Q Ai and key K Bj Attention weights between them; Based on the feature uncertainty U corresponding to the j-th eigenvector in the second latent space representation Bj The value vector V obtained by performing another linear transformation on the j-th eigenvector in the second latent space representation. Bj Scale the vector to obtain the adjusted value vector V′. Bj ; In the attention mechanism, the calculated attention weights are used to adjust the value vector V′. Bj Perform a weighted summation to obtain the result for query Q. Ai Context-related feature vectors; Will target query Q Ai Context-related feature vectors and query Q Ai By combining preset fusion operations, fused features are generated.

13. The method according to claim 1, characterized in that, Registration parameters for aligning the preoperative and intraoperative medical images, based on the fused feature estimation, include: The fused features are input into a deep neural network with a VoxelMorph-like U-Net architecture, and the deep neural network is used to regress the dense displacement field as the registration parameter. The deep neural network is optimized using a composite loss function, which includes semantic similarity loss and regularization loss. The semantic similarity loss is obtained by calculating the similarity between the features obtained by deforming the first latent space representation according to the estimated dense displacement field and the second latent space representation in the target latent space. The regularization loss is obtained by calculating the norm of the gradient of the estimated dense displacement field or its bending energy term.

Images