Microsurgical robot ultrasonic fusion multi-modal image guided surgery navigation system

Abstract

The present application relates to the technical field of medical devices, and discloses an ultrasonic fusion multi-modal image guided surgery navigation system of a microsurgical robot. The system comprises system initialization, multi-modal image acquisition, real-time data monitoring, image fusion analysis, navigation stability evaluation and adaptive navigation adjustment modules. The system initialization module receives planning parameters and configures a navigation reference; the multi-modal image acquisition module synchronously controls ultrasonic and complementary image equipment to acquire data, and outputs a calibrated image set after alignment verification; the real-time data monitoring module tracks the trajectory and attitude of the instrument, and generates dynamic data. The image fusion analysis module completes image space transformation and superposition, and evaluates fusion consistency; the navigation stability evaluation module calculates deviation tolerance; and the adaptive navigation adjustment module optimizes the navigation path accordingly. The system integrates multi-source images and instrument dynamic information, improves navigation accuracy and stability, and is suitable for fine microsurgical operations.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to an ultrasonic fusion multi-modal image guided surgery navigation system for microsurgical robots. BACKGROUND

[0002] In microsurgery, a surgery navigation system is a key tool to assist surgeons in precise operations. In the prior art, most navigation systems rely on a single image modality, such as using only ultrasonic images or CT images. Such systems are difficult to fully capture the anatomical details of the surgical area and cannot balance real-time dynamic information and high-resolution structural information. At the same time, the tracking of surgical instruments in traditional systems is mostly limited to static position recording, lacking continuous dynamic analysis of instrument motion trajectories and attitude angles, and is difficult to reflect subtle changes in instruments during surgery in real time.

[0003] The prior art has obvious limitations. The information dimension provided by a single image modality is limited, which can easily lead to deviations in the judgment of the relationship between the lesion and the surrounding tissue by the surgeon. The dynamic nature of instrument tracking is insufficient, making the navigation feedback lag behind the actual operation and affecting the accuracy of the operation. In addition, the path planning parameters of traditional navigation systems are fixed and do not consider the impact of image quality fluctuations and intraoperative environmental changes on navigation stability. When the fused images deviate or the instrument motion exceeds the expected range, the system cannot adjust itself and may exacerbate the operation error.

[0004] With the increasing precision requirements of microsurgery, more comprehensive image information support is needed to clearly present complex anatomical structures. At the same time, the navigation system needs to respond to instrument dynamics and image quality changes in real time to avoid navigation failure caused by fixed parameters. This requires breaking through the limitations of a single modality, achieving precise fusion of multi-source images, and establishing a dynamic navigation adjustment mechanism based on real-time data to address the problems of incomplete information and insufficient adaptability that cannot be solved by existing technology. SUMMARY

[0005] The purpose of the present application is to provide an ultrasonic fusion multi-modal image guided surgery navigation system for microsurgical robots to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides an ultrasonic fusion multi-modal image guided surgery navigation system for microsurgical robots, which comprises:

[0007] A system initialization module receives surgery planning parameters and configures navigation reference values, and generates a system activation signal;

[0008] A multi-modal image acquisition module controls an ultrasonic imaging device and a complementary image device to acquire real-time anatomical data based on the system activation signal, performs multi-source image alignment and quality verification, and outputs a calibrated multi-modal image set.

[0009] a real-time data monitoring module configured to track a motion trajectory and an attitude angle of the surgical instrument according to the multi-modal image set, analyze a position change trend of the instrument, and generate instrument dynamic tracking data;

[0010] an image fusion analysis module configured to perform spatial transformation and content superposition of the ultrasound and multi-modal images based on the instrument dynamic tracking data and the multi-modal image set, evaluate fusion consistency, and generate fusion image results and fusion quality parameters;

[0011] a navigation stability evaluation module configured to calculate a deviation tolerance of the navigation system using the fusion image results and fusion quality parameters and the instrument dynamic tracking data, and output a stability evaluation report;

[0012] an adaptive navigation adjustment module configured to adjust a curve smoothness and a fault tolerance range of the navigation path according to the stability evaluation report, and generate optimized navigation instructions.

[0013] Preferably, the multi-modal image acquisition module outputs the calibrated multi-modal image set by:

[0014] synchronously triggering the ultrasound probe and the optical coherence tomography instrument to capture a raw image stream, recording a time stamp and device parameters of each image frame, performing multi-scale filtering processing on the raw image stream to reduce noise interference, and using a contrast-limited adaptive histogram equalization method to enhance image details, extracting edge intensity and texture uniformity features of the enhanced image, calculating an image definition index, comparing the image definition index with a preset definition threshold, and if the definition index is lower than the threshold, automatically adjusting device focusing parameters or reacquiring images, integrating all qualified image frames, and arranging them in chronological order to generate the multi-modal image set.

[0015] Preferably, the image fusion analysis module generates the fusion image results and fusion quality parameters by:

[0016] performing scale-invariant feature transform detection on the ultrasound image and the multi-modal image to generate a set of key point descriptors, matching the key point descriptors using a random sample consensus algorithm to estimate an affine transformation matrix between the images, applying the affine transformation matrix to map the ultrasound image to the multi-modal image coordinate system, calculating a pixel alignment error, analyzing the consistency degree of the image overlap region based on the pixel alignment error to generate a fusion quality score, and using a weighted average method to fuse the pixel values of the aligned images to output the fusion image results.

[0017] Preferably, the image fusion analysis module further evaluates the fusion consistency by:

[0018] The fusion image results of continuous time sequence are acquired to construct an image stability time sequence; the time sequence is subjected to empirical mode decomposition to obtain intrinsic mode function components; sample entropy values of each component are calculated to quantify complexity; a fluctuation range of the sample entropy values is analyzed to derive an image time sequence stability index; a comprehensive fusion quality parameter is generated by combining the pixel alignment error and the time sequence stability index.

[0019] Preferably, the image fusion analysis module generates a fusion quality parameter in the following steps:

[0020] The pixel alignment error and the image time sequence stability index are normalized into a feature vector; a decision tree model is used to perform regression analysis on the feature vector to predict a fusion quality level; and a specific value of the fusion quality parameter is output according to a quality level mapping table.

[0021] Preferably, the navigation stability evaluation module outputs a stability evaluation report in the following steps:

[0022] The fusion quality parameter and instrument dynamic tracking data are received; a multi-level quality tolerance threshold is set to divide the fusion quality parameter into high consistency, medium consistency and low consistency levels; for each level, a navigation deviation coefficient is calculated in combination with instrument position fluctuation; and system reliability is evaluated based on the navigation deviation coefficient to generate a stability evaluation report.

[0023] Preferably, the real-time data monitoring module generates instrument dynamic tracking data in the following steps:

[0024] Three-dimensional coordinate and Euler angle data of the instrument are continuously collected; an extended Kalman filter algorithm is applied to smooth and predict the data to estimate the real position of the instrument; a residual sequence of the observed position and the predicted position is calculated; a root mean square error of the residual sequence is analyzed to obtain a position drift index; a calibration program is triggered when the position drift index exceeds a dynamic threshold, otherwise the normal tracking state is maintained; and instrument dynamic tracking data containing the position and the drift index are output.

[0025] Preferably, the real-time data monitoring module further adaptively adjusts the sampling strategy according to the position drift index: when the position drift index increases, the sensor sampling frequency is increased and redundant sensor data fusion is enabled; when the position drift index decreases, the default sampling settings are restored to optimize resource usage.

[0026] Preferably, the adaptive navigation adjustment module generates optimized navigation instructions in the following steps:

[0027] The navigation deviation coefficient in the stability evaluation report and the position drift index in the instrument dynamic tracking data are analyzed, an adaptive sliding mode control algorithm is used to calculate a path correction amount, wherein a control gain dynamically changes according to the deviation coefficient, the correction amount is applied to a current navigation path control point to generate a new path with smooth transition, and a coordinate sequence of the new path is output as an optimized navigation instruction.

[0028] Preferably, the training and application steps of the decision tree model are specifically as follows:

[0029] A historical feature vector dataset containing true fusion quality grades is collected, a decision tree model is generated on the training set using a CART algorithm, and a pruning strategy is used to avoid overfitting;

[0030] In the decision tree model application stage, the real-time normalized feature vector is subjected to conditional judgment from the root node of the decision tree to the leaf node, and the fusion quality grade prediction result corresponding to the leaf node is output, and the probability confidence of belonging to the grade is calculated.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] The multi-modal image acquisition module controls the ultrasound imaging device and the complementary image device to synchronously collect real-time anatomical data, and after multi-source image alignment and quality verification, a calibrated multi-modal image set is output, and then the image fusion analysis module performs spatial transformation and content superposition of ultrasound and multi-modal images based on instrument dynamic tracking data. This technical solution fuses the real-time dynamic imaging capability of ultrasound with the high-resolution structural information of other images, breaking through the limitations of a single image modality in terms of tissue detail presentation and dynamic change capture. Ultrasound can reflect the dynamic displacement of intraoperative tissues in real time, while complementary images can provide clear anatomical structure baseline. After spatial transformation and content superposition, the static anatomical framework and dynamic tissue changes can be presented simultaneously, making the information presentation of the surgical area more complete, allowing the surgeon to grasp the overall structural relationship and perceive real-time dynamic changes during operation, avoiding judgment bias caused by one-sided information.

[0033] The navigation stability evaluation module calculates the deviation tolerance by using the fusion image result, the fusion quality parameter and the instrument dynamic tracking data and outputs an evaluation report, and the adaptive navigation adjustment module adjusts the curve smoothness and the fault tolerance range of the navigation path according to the evaluation report. The technical scheme calculates the deviation tolerance in real time, dynamically perceives the stability state of the navigation system in the complex intraoperative environment. When the fusion image quality fluctuates or the instrument motion appears a slight deviation, the system can adjust the curve smoothness of the path, reduce the operation jam caused by the too strong rigidity of the path, and adjust the fault tolerance range to adapt to the dynamic change within a reasonable range on the premise of ensuring the accuracy. The dynamic adjustment mechanism makes the navigation path match the actual situation in real time, avoids the continuous accumulation of errors under the fixed parameters, makes the instrument motion and the navigation guidance more closely coordinated, and meets the requirements of the fine operation in the microsurgery on the flexibility and stability of the navigation. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A working principle diagram of the microsurgical robot ultrasonic fusion multi-modal image guided surgery navigation system is described.

[0035] Figure 2 A flowchart of image fusion and quality evaluation is described.

[0036] Figure 3 A flowchart of fusion consistency timing stability evaluation is described.

[0037] Figure 4 An image fusion quality feature distribution and grade division diagram is described.

[0038] Figure 5 A surgery navigation system stability evaluation analysis diagram is described. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0040] Please refer to Figure 1The application provides a microsurgical robot ultrasonic fusion multi-modal image guided surgery navigation system, which comprises a system initialization module, a multi-modal image acquisition module, a real-time data monitoring module, an image fusion analysis module, a navigation stability evaluation module and an adaptive navigation adjustment module.

[0041] Example 1: see Figure 2In specific implementation, the multi-modal image acquisition module initiates workflow through system activation signal generated by the system initialization module, synchronously triggers the ultrasound probe and the optical coherence tomography scanner to capture the raw image stream, the raw image stream contains continuous two-dimensional or three-dimensional image sequence, and the time stamp and device parameters of each image frame are recorded, the device parameters include the gain, frequency and focal length settings of the ultrasound imaging device and the scanning depth and resolution parameters of the optical coherence tomography scanner. In specific implementation, the raw image stream is subjected to multi-scale filtering processing to reduce noise interference, the multi-scale filtering processing adopts a Gaussian pyramid decomposition method, applies a difference filter on different resolution levels to highlight image features and suppress random noise, and adopts a contrast-limited adaptive histogram equalization method to enhance image details, the contrast-limited adaptive histogram equalization method avoids over-enhancement by dividing the image into multiple local regions and limiting the histogram distribution of each region. In specific implementation, the edge intensity and texture uniformity features of the enhanced image are extracted, the edge intensity feature uses a Sobel operator to calculate the gradient amplitude, and the texture uniformity feature is derived based on a gray level co-occurrence matrix, so as to calculate an image sharpness index, the image sharpness index is a weighted combination value of the edge intensity and the texture uniformity. In specific implementation, the image sharpness index is compared with a preset sharpness threshold, the preset sharpness threshold is pre-set according to the clinical application scene, if the image sharpness index is lower than the preset sharpness threshold, the device focusing parameters are automatically adjusted or the image is re-acquired, when the device focusing parameters are adjusted, the physical settings of the ultrasound probe or the optical coherence tomography scanner are optimized through a closed-loop control algorithm. In specific implementation, all qualified image frames are integrated, arranged in time sequence to generate a multi-modal image set, and the multi-modal image set ensures time sequence consistency and spatial alignment basis. It can be understood that the entire process of the multi-modal image acquisition module relies on hardware trigger synchronization and software processing pipeline to achieve efficient data acquisition and quality control.

[0042] In specific implementation, the image fusion analysis module receives the multi-modal image set output by the multi-modal image acquisition module and the instrument dynamic tracking data provided by the real-time data monitoring module, and starts to perform fusion analysis. The image fusion analysis module first performs scale-invariant feature transform detection on the ultrasound image and the multi-modal image. The scale-invariant feature transform detection algorithm identifies key points in the image and generates a set of key point descriptors, which contain position, scale and direction information. In specific implementation, the random sample consensus algorithm is used to match the key point descriptors. The random sample consensus algorithm estimates the affine transformation matrix between images through iterative random sampling and consistency test, and the affine transformation matrix describes the rotation, translation and scaling relationship. In specific implementation, the affine transformation matrix is applied to map the ultrasound image to the multi-modal image coordinate system, and the pixel alignment error is calculated. The pixel alignment error is obtained by comparing the pixel distance of the overlapping area. In specific implementation, the consistency degree of the image overlapping area is analyzed based on the pixel alignment error, and a fusion quality score is generated. The fusion quality score is calculated by using the weighted average method combining the error value and the area overlap rate. In specific implementation, the weighted average method is used to fuse the pixel values of the aligned images, and the weight is dynamically adjusted according to the image signal-to-noise ratio. The fusion image result is output. It can be understood that the fusion process of the image fusion analysis module emphasizes the accuracy of geometric alignment and content superposition, providing reliable input for subsequent navigation.

[0043] In some embodiments, the synchronization trigger mechanism of the multi-modal image acquisition module adopts a hardware signal synchronization method. The ultrasound probe and the optical coherence tomography instrument coordinate the acquisition timing through an external clock signal to ensure that the timestamps of the original image streams correspond accurately. In some embodiments, the number of Gaussian pyramid decomposition layers of the multi-scale filtering process can be adaptively selected according to the image resolution. For example, more pyramid levels are used for high-resolution images to preserve details. Optionally, the contrast limit parameter in the contrast limited adaptive histogram equalization method can be adjusted according to the image type. For example, a more stringent limit value is used for low-contrast images. Optionally, the calculation of the image sharpness index can introduce a multi-feature fusion algorithm, combining edge strength, texture uniformity and signal-to-noise ratio indicators. It can be understood that the quality verification step of the multi-modal image acquisition module is the basis for ensuring the reliability of subsequent fusion.

[0044] In some embodiments, the scale-invariant feature transform detection of the image fusion analysis module can be extended to a multi-modal adaptation version, optimizing feature extraction parameters for different characteristics of ultrasound images and multi-modal images. In some embodiments, the iteration number and error threshold of the random sample consensus algorithm can be dynamically set to improve matching robustness. Optionally, the calculation of pixel alignment error can include local region error distribution analysis to identify areas of alignment inconsistency. Optionally, when fusing pixel values using the weighted average method, the weight distribution strategy can be adjusted based on the clinical importance of the image region, for example, giving higher weight to key anatomical structures. It can be understood that the fusion quality parameter generation process of the image fusion analysis module integrates multi-dimensional evaluation to support navigation stability judgment.

[0045] In specific implementation, the image sharpness index comparison step of the multi-modal image acquisition module integrates automatic decision logic, when the sharpness index is below the threshold, the system preferentially tries to adjust the device focus parameters, such as changing the focal length of the ultrasound probe or the scanning depth of the optical coherence tomography scanner, if it still does not meet the standard after adjustment, it triggers the reacquisition program. In specific implementation, the generation of the multi-modal image set includes a data format standardization step, which converts image frames of different sources into a unified coordinate system and pixel format, facilitating subsequent processing. In specific implementation, the key point matching stage of the image fusion analysis module uses a multi-resolution strategy, first matching quickly in low-resolution images and then refining to high-resolution to improve efficiency. In specific implementation, the generation of the fusion quality score uses a fuzzy logic system to map pixel alignment error and overlap rate to score values. It can be understood that the implementation of the entire embodiment 1 emphasizes the coordination of data flow between modules and real-time processing capability.

[0046] In specific implementation, the device parameter record of the multi-modal image acquisition module includes storing context information of the acquisition environment, such as patient position and device posture, for subsequent retrospective analysis. In specific implementation, the noise interference reduction effect of multi-scale filtering processing is quantified by signal-to-noise ratio improvement index, but the system does not rely on this data for real-time decision making. In specific implementation, the affine transformation matrix estimation of the image fusion analysis module contains an outlier rejection mechanism, using the inlier set of the random sample consensus algorithm to optimize transformation accuracy. In specific implementation, when fusing pixel values using the weighted average method, the weight ratio of ultrasound images and multi-modal images can be dynamically adjusted according to the fusion quality score. It can be understood that the implementation details of embodiment 1 aim to realize a repeatable and robust image acquisition and fusion process.

[0047] In practical implementation, the re-acquisition procedure of the multimodal image acquisition module includes a maximum retry limit to avoid infinite loops, and the system issues an alarm when a retry times out. In practical implementation, the pixel alignment error calculation of the image fusion analysis module uses Euclidean distance to calculate the mapped positional deviation for each pixel. In practical implementation, the output format of the fused image results includes multi-layer image stacking, preserving the original data channels for visualization.

[0048] Example 2: See Figure 3 In practical implementation, the image fusion analysis module acquires fused image results from continuous time series. These results are derived from the continuous fusion output of the module, constructing an image stability time series. This time series extracts pixel intensity statistics for specific regions from the fused image results at fixed sampling intervals. Furthermore, empirical mode decomposition (EMD) is performed on the image stability time series. EMD decomposes the time series signal into a series of intrinsic mode function (EMF) components through an iterative selection process. Each EMF component satisfies the condition that the number of local extrema and zero-crossings differs by no more than one, and the local mean is determined by averaging the upper and lower envelopes. Finally, the sample entropy value of each EMF component is calculated to quantify complexity. A fixed embedding dimension *m* and tolerance parameter *r* are used in the sample entropy calculation, and the irregularity of the time series is measured by the probability of statistical template vector matching. In practical implementation, the fluctuation range of sample entropy values ​​is analyzed. This fluctuation range is assessed by calculating the standard deviation and range of the sample entropy values ​​of all intrinsic mode function components, thereby deriving the image temporal stability index, which is a weighted combination of the standard deviation and range. In practice, a comprehensive fusion quality parameter is generated by combining pixel alignment error and the image temporal stability index. The combination process employs a linear weighted combination method, with weighting coefficients preset based on the relative importance of spatial alignment accuracy and temporal stability in clinical applications. It can be understood that the temporal stability analysis of the image fusion analysis module focuses on capturing the dynamic characteristics of the fusion results in the time dimension, providing a more comprehensive evaluation basis for navigation stability.

[0049] In implementations, the construction of the image stability time series requires defining a fixed region of interest, which is usually delineated around the tip of a surgical instrument or a critical anatomical structure, and extracting the mean or variance of pixel intensity of this region from each fused image result frame as a data point of the time series. In implementations, the iterative sifting process of empirical mode decomposition involves finding all local extrema of the signal, fitting upper and lower envelope curves with cubic spline interpolation respectively, calculating the mean of the upper and lower envelope curves and subtracting this mean from the original signal to obtain a candidate intrinsic mode function component, repeating the above steps until the candidate intrinsic mode function component meets the stopping criterion. In implementations, the calculation of the sample entropy value involves constructing template vectors of length m and m+1, and counting the proportion of template vectors that match in the entire time series within a tolerance r, with the sample entropy value defined as the negative natural logarithm result. In implementations, the derivation formula of the image temporal stability indicator is a linear combination, for example a standard deviation + b range, where the weights a and b are preset constants. In implementations, the generation of the comprehensive fusion quality parameter adds the normalized pixel alignment error and the image temporal stability indicator, possibly introducing a scaling factor to map the result to a preset quality score range. It can be understood that the entire evaluation process relies on the systematic application of signal processing techniques, which converts the spatiotemporal characteristics of the image into quantifiable stability measures.

[0050] In some embodiments, the sampling interval of the image stability time series can be adjusted according to the real-time requirements of the system, for example, a shorter sampling interval is used in high dynamic surgical stages to capture more subtle temporal changes. In some embodiments, the stopping criterion of empirical mode decomposition can use the standard deviation criterion, that is, when the standard deviation of the results of two consecutive screenings is less than a preset threshold, the iteration is stopped. Optionally, the embedding dimension m of the sample entropy value calculation parameter is usually 2, and the tolerance parameter r is usually 0.1 to 0.25 times the standard deviation of the time series. Optionally, the weight allocation of the standard deviation and the range in the image temporal stability indicator can be customized according to the sensitivity of different surgical types to short-term fluctuations and long-term drifts. It can be understood that the parameter settings of the temporal stability analysis module need to strike a balance between computational complexity and evaluation accuracy.

[0051] In some embodiments, in addition to pixel intensity statistics, other image features such as texture feature values computed from the fusion region can be incorporated into the construction of the image stability time series to build a multi-dimensional time series for more comprehensive stability evaluation. In some embodiments, the residual term produced in the empirical mode decomposition process, i.e. the trend term, can also be used for stability analysis, for example by computing the slope of the trend term to evaluate the long-term drift characteristics of the fusion result. Optionally, the sample entropy values can be computed separately for different intrinsic mode function components at different time scales, and then the distribution pattern of the sample entropy values across different scales can be analyzed. Optionally, the combination of pixel alignment error and image time series stability indicator can be non-linear, for example by using a rule-based system or lookup table to map the combination of the two to the final comprehensive fusion quality parameter. It can be appreciated that the implementation of embodiment 2 provides a concrete and quantifiable technical path for evaluating fusion consistency.

[0052] In a specific implementation, the image fusion analysis module contains a dedicated time series data processing unit that is responsible for managing the buffer of the image stability time series, and performing the empirical mode decomposition and sample entropy computation algorithms in a time series manner. In a specific implementation, the implementation of the empirical mode decomposition algorithm requires handling of boundary effect issues, and typically uses mirror extension or characteristic wave extension methods to extend the signal at both ends to reduce decomposition errors. In a specific implementation, the computation process of the sample entropy value needs to be optimized to avoid numerical calculation underflow, and typically a small positive number epsilon is added when calculating the probability to ensure numerical stability. In a specific implementation, the calculation of the image time series stability indicator is periodic, and its update frequency is consistent with or an integer multiple of the construction frequency of the image stability time series.

[0053] In a specific implementation, the number of intrinsic mode function components depends on the length and complexity of the original image stability time series, and the system will pre-allocate sufficient memory space to store all the decomposed components. In a specific implementation, the fluctuation range analysis of the sample entropy value will be performed once in each evaluation period, and the system will maintain a sliding window of historical sample entropy values for calculating the fluctuation statistics of the current period. In a specific implementation, the output of the comprehensive fusion quality parameter is a real-time numerical stream, which is sent to the navigation stability evaluation module together with the fusion image result for its use. It can be appreciated that the introduction of the time series stability indicator enables the system to perceive those slow-accumulating fusion quality changes that are difficult to detect in single-frame analysis, thereby improving the forward-looking and robustness of the navigation system.

[0054] In a specific implementation, the image fusion analysis module normalizes the pixel alignment error and the image temporal stability index into a feature vector, the normalization process linearly transforms the original values of the pixel alignment error and the image temporal stability index into the interval [0, 1] using the min-max scaling method, the feature vector is a two-dimensional vector, the first element of which corresponds to the normalized pixel alignment error value, and the second element corresponds to the normalized image temporal stability index value. In a specific implementation, the decision tree model is used to perform regression analysis on the feature vector, the decision tree model is a prediction model trained on the historical data set based on the CART algorithm, and the regression analysis process is to input the feature vector and output a continuous fusion quality level prediction value. In a specific implementation, the specific value of the fusion quality parameter is output according to the quality level mapping table, and the quality level mapping table is a predefined lookup table that maps the discrete fusion quality level predicted by the decision tree model to a numerical score with clear physical meaning. It can be understood that combining the pixel alignment error and the image temporal stability index into a feature vector and performing regression analysis through a decision tree model can capture the nonlinear relationship between the two indicators, thereby generating a more accurate fusion quality parameter.

[0055] The training and application steps of the decision tree model include collecting a historical feature vector data set containing labeled real fusion quality levels, each sample in the historical feature vector data set contains a historical feature vector and a real fusion quality level labeled by an expert. In a specific implementation, the CART algorithm is used to generate a decision tree model on the training set, the CART algorithm splits nodes by recursively selecting features and dividing points, the selection criterion is the minimization of squared error, and the pruning strategy is used to avoid overfitting, the pruning strategy adopts the cost complexity pruning, which decides the optimal subtree by minimizing the composite loss function

[0056]

[0057] wherein: represents the decision tree model the sum of the mean square errors of the sample output values of all leaf node regions on the training data, represents the decision tree model the number of leaf nodes of the decision tree model is a hyperparameter, and the optimal value is selected by cross-validation to balance the goodness of fit and the complexity of the model.

[0058] ​In the decision tree model application stage, the real-time normalized feature vector is subjected to conditional judgment from the root node of the decision tree to the leaf node. The conditional judgment is based on the feature index and the division threshold stored on each internal node of the decision tree model. In specific implementation, the fusion quality level prediction result corresponding to the leaf node is output, and the probability confidence that it belongs to the level is calculated. The probability confidence is calculated based on the proportion of samples belonging to the predicted level in the training samples contained in the leaf node in the training stage. It can be understood that the application of the decision tree model realizes the automatic and interpretable mapping from multi-dimensional features to a single quality level.

[0059] In some embodiments, the normalization process of the feature vector can adopt other standardization methods, such as Z-score standardization, to convert the pixel alignment error and the image timing stability index into a distribution with a mean of 0 and a standard deviation of 1. In some embodiments, the regression analysis of the decision tree model can output a continuous numerical value rather than a discrete level, and at this time the role of the quality level mapping table is to discretize the continuous numerical value into the preset level interval. In some embodiments, when the CART algorithm generates the decision tree model, the splitting criterion in the classification task can adopt Gini impurity minimization. Optionally, the hyperparameter a in the cost complexity pruning can be determined by grid search on the validation set through cross-validation to determine the optimal value. It can be understood that the training process of the decision tree model depends on high-quality historical labeled data, and the quality directly affects the accuracy of the final prediction.

[0060] In specific implementation, the construction of the historical feature vector dataset is an important prerequisite, which requires collecting a large amount of pixel alignment error and image timing stability index data obtained under different surgical scenarios and imaging conditions, and labeling the fusion quality level by multiple experienced physicians according to unified standards. In specific implementation, the training set and the test set of the decision tree model need to be randomly divided, and the historical feature vector dataset is usually divided according to a ratio of 7:3 or 8:2 to ensure the fairness of model evaluation. In specific implementation, when the decision tree model traverses the nodes in the application stage, the first element and the second element of the feature vector are compared in turn, and according to the comparison result, it is decided to enter the left subtree or the right subtree. In specific implementation, the calculation of the probability confidence provides an uncertainty measure for the real-time navigation process. When the probability confidence is low, the system can trigger a warning to remind the operator to pay attention to the current fusion quality evaluation result, which may not be reliable. It can be understood that the generation process of the fusion quality parameter combines quantitative image features with trained machine learning models to realize intelligent evaluation of fusion consistency.

[0061] In a specific implementation, the structure of the decision tree model is stored in the system as a file in XML or JSON format, which is loaded into memory when the image fusion analysis module is initialized to facilitate fast prediction. In a specific implementation, in order to avoid the decision tree model being outdated, the system is designed with a model updating mechanism, which can periodically retrain or fine-tune the decision tree model using new surgical data. In a specific implementation, the content of the quality level mapping table is configurable, allowing medical institutions to adjust the correspondence between fusion quality levels and numerical scores according to specific clinical needs and safety standards. It can be understood that the embodiments of Embodiment 3 describe in detail how to convert the underlying image processing indicators into high-level quality parameters that can be directly used for navigation decisions through a data-driven model.

[0062] Referring to Figure 4 , which shows the image fusion quality analysis results based on the decision tree model. In the figure, three different consistency level regions are marked with different colors, corresponding to low consistency, medium consistency and high consistency states respectively. Each scatter point represents a feature vector of a time sampling point, the horizontal coordinate represents the pixel alignment error indicator, which reflects the accuracy of multi-modal image in the spatial registration process; the vertical coordinate represents the image temporal stability indicator, which measures the fluctuation of image fusion quality in the continuous time sequence. The decision tree model can effectively identify the quality change trend in the image fusion process by analyzing the combination of these two key features, providing reliable quality evaluation basis for surgical navigation. The figure clearly shows the distribution law of different quality levels, which helps the operator to intuitively understand the fusion performance state of the current system.

[0063] In a specific implementation, the navigation stability evaluation module receives the fusion quality parameter from the image fusion analysis module and the instrument dynamic tracking data from the real-time data monitoring module, and sets multiple levels of quality tolerance thresholds inside the navigation stability evaluation module, including the upper limit of the high consistency level threshold, the lower limit of the high consistency level threshold, the upper limit of the medium consistency level threshold, the lower limit of the medium consistency level threshold, and the low consistency level threshold, which divide the continuous fusion quality parameter values into three discrete levels of high consistency, medium consistency, and low consistency. In a specific implementation, for each consistency level, the navigation stability evaluation module calculates the navigation deviation coefficient by combining the instrument position fluctuation contained in the instrument dynamic tracking data, and the navigation deviation coefficient is calculated by taking the weighted sum of the normalized fusion quality parameter reciprocal and the instrument position fluctuation, and its calculation formula is in the form of linear combination. In a specific implementation, based on the navigation deviation coefficient to evaluate the system reliability, a stability evaluation report is generated, which contains the current navigation deviation coefficient value, the corresponding confidence level, and the system operation suggestion. It can be understood that the navigation stability evaluation module combines the image fusion quality and the instrument motion state to realize the quantitative evaluation of the overall stability of the navigation system.

[0064] The specific steps of the real-time data monitoring module to generate instrument dynamic tracking data include continuously collecting three-dimensional coordinate and Euler angle data of the surgical instrument, three-dimensional coordinate data is obtained through an optical positioning system, and Euler angle data is obtained through an inertial measurement unit fixed on the instrument. In a specific implementation, the collected three-dimensional coordinate and Euler angle data are smoothed and predicted by applying an extended Kalman filter algorithm to estimate the true position and attitude of the surgical instrument, and the extended Kalman filter algorithm realizes optimal estimation by establishing a nonlinear state space model of instrument motion and recursively performing prediction and update steps. In a specific implementation, the residual sequence of the observed position and the predicted position by the extended Kalman filter algorithm is calculated, and the residual sequence is a time sequence composed of the difference between the observed value and the predicted value at each sampling time. In a specific implementation, the root mean square error of the residual sequence is analyzed to obtain the position drift index, which is a long-term statistic that measures the deviation between the actual motion of the instrument and the model prediction. In a specific implementation, the calibration program is triggered when the position drift index exceeds the dynamic threshold, otherwise the normal tracking state is maintained, and the dynamic threshold is self-adaptively adjusted according to the historical statistics of the recent position drift index. In a specific implementation, complete instrument dynamic tracking data containing instrument position, attitude, velocity, and position drift index are output. It can be understood that the real-time data monitoring module ensures the accuracy and reliability of the instrument tracking data through advanced filtering algorithms and drift detection mechanisms. Referring to Table 1, the multi-level quality tolerance threshold settings of the navigation stability evaluation module can be configured with reference to Table 1.

[0065] Table 1: Fusion quality parameter level division threshold

[0066]

[0067] In specific implementations, the calculation of the navigation deviation coefficient employs different weight factors for each consistency level, for example, at a high consistency level, the weight of the fusion quality parameter is higher, and the weight of the instrument position fluctuation is lower, while at a low consistency level, a higher weight is given to the instrument position fluctuation. In specific implementations, the reliability level of the stability evaluation report is divided into three levels of high reliability, medium reliability, and low reliability, corresponding to three preset intervals of the navigation deviation coefficient. In specific implementations, the state vector of the extended Kalman filter algorithm includes the position, velocity, acceleration, and Euler angle and angular velocity of the instrument, and the process noise covariance matrix and the observation noise covariance matrix need to be pre-calibrated according to the motion characteristics of the instrument. In specific implementations, the calculation of the position drift index adopts a sliding window method, and the window length is usually set to 100-200 sampling points to balance the real-time performance of the calculation and the stability of the statistics.

[0068] In some embodiments, the multi-level quality tolerance threshold can be a fixed value, or can be dynamically adjusted according to different stages of the surgery or different target anatomical structures, for example, a stricter threshold is used in the fine operation stage. Optionally, the calculation of the navigation deviation coefficient can introduce a more complex nonlinear function, for example, based on a neural network model to map the fusion quality parameter and the instrument position fluctuation to the deviation coefficient. Optionally, the system operation suggestion of the stability evaluation report can include specific instructions such as "continue operation", "prompt warning", or "suggest to pause calibration". It can be understood that the output of the navigation stability evaluation module is an important basis for the decision of the adaptive navigation adjustment module.

[0069] In specific implementations, the root mean square error calculation of the residual sequence adopts the standard formula, that is, first calculate the sum of squares of the residual sequence, then divide by the sequence length and take the square root. In specific implementations, the trigger mechanism of the calibration program contains a debounce logic, that is, the position drift index is required to exceed the dynamic threshold for a plurality of consecutive sampling periods to confirm the trigger, in order to avoid false calibration caused by accidental fluctuations. In specific implementations, the instrument dynamic tracking data is output at a fixed frequency and format, usually using a structure data format to encapsulate all tracking information. In specific implementations, the adaptive adjustment of the dynamic threshold is based on the mean and standard deviation of the position drift index in the past period of time, for example, the dynamic threshold can be set to twice the historical mean plus three times the historical standard deviation. It can be understood that the drift detection and calibration mechanism of the real-time data monitoring module is the key guarantee for maintaining long-term tracking accuracy.

[0070] In specific implementations, the prediction step of the extended Kalman filter algorithm uses a linearized state transition matrix to approximate the nonlinear system dynamics, whose Jacobian matrix needs to be calculated online or retrieved from a pre-computed lookup table. In specific implementations, the instrument dynamic tracking data contains not only the state estimate at the current time instant, but also the covariance matrix information of the state, which is used to represent the uncertainty of the estimate. In specific implementations, the navigation stability evaluation module and the real-time data monitoring module are usually run as independent software threads or processes, with high-speed data exchange through shared memory or message queues.

[0071] Referring to Figure 5 , the comprehensive analysis results of the navigation system stability evaluation are presented. The chart shows the trend of the navigation bias coefficient at different time points, and divides the three working areas of high reliability, medium reliability and low reliability through three horizontal threshold lines. The navigation bias coefficient is calculated by the weighted combination of the inverse of the fusion quality parameter and the instrument position fluctuation, which can comprehensively reflect the overall stability of the system in the current state. When the bias coefficient is in the green area, it indicates that the system is stable and reliable; when it enters the orange area, it prompts to pay attention to the system state; when it reaches the red area, it is recommended to calibrate or adjust the system. This evaluation mechanism combines the dual information of image fusion quality and instrument motion state, providing an important quantitative reference for navigation decision-making during surgery, ensuring the safety and accuracy of surgical operations. The time series changes in the chart reflect the stability performance of the system at different stages of surgery.

[0072] In specific implementations, the real-time data monitoring module continuously monitors the position drift index calculated by the extended Kalman filter algorithm, which is a dimensionless value reflecting the degree of deviation between the actual motion trajectory of the surgical instrument and the prediction model. When the position drift index rises and exceeds the first preset threshold, the real-time data monitoring module automatically increases the sampling frequency of sensors such as optical positioning systems and inertial measurement units, for example, from the default 100 Hz to 200 Hz, while enabling the redundant sensor data fusion strategy, which means that data from multiple sensors of the same type (such as two inertial measurement units) are weighted and averaged or Kalman filtered to reduce random errors and transient disturbances of a single sensor. When the position drift index decreases below the second preset threshold, the real-time data monitoring module restores the sensor sampling frequency to the default 100 Hz setting and closes the redundant sensor data fusion function to optimize computing resources and system power consumption. It can be understood that this adaptive sampling strategy based on the position drift index can improve the overall energy efficiency of the system while ensuring tracking accuracy.

[0073] In specific implementation, the process of the adaptive navigation adjustment module generating the optimized navigation instruction starts from analyzing the stability evaluation report output by the navigation stability evaluation module and the instrument dynamic tracking data provided by the real-time data monitoring module, the adaptive navigation adjustment module extracts the navigation deviation coefficient from the stability evaluation report and reads the position drift index from the instrument dynamic tracking data, and calculates the path correction amount by using the adaptive sliding mode control algorithm. The adaptive sliding mode control algorithm is a nonlinear control method, the core of which is to design a sliding mode surface and construct a control law to make the system state trajectory be attracted to the sliding mode surface in a limited time. In the adaptive sliding mode control algorithm, the size of the control gain dynamically changes according to the navigation deviation coefficient. When the navigation deviation coefficient is large, a larger control gain is used to achieve rapid correction, and when the navigation deviation coefficient is small, a smaller control gain is used to maintain motion stability. In specific implementation, the calculated path correction amount is applied to the control point sequence of the current navigation path. The control point is a key coordinate for defining the B-spline navigation path. The modified control points are fitted by using the B-spline curve interpolation algorithm to generate a new path with smooth transition. The new path is geometrically guaranteed to be second-order continuous and differentiable, avoiding sudden changes and jitter in motion. In specific implementation, the coordinate sequence of the new path is output as the optimized navigation instruction. The coordinate sequence is stored in an array form, including a series of discrete point coordinates and their corresponding attitude quaternions in three-dimensional space.

[0074] In some embodiments, the judgment of the position drift index can use a multi-threshold hysteresis comparison strategy, that is, the threshold for triggering the increase of the sampling frequency is higher than the threshold for triggering the recovery of the sampling frequency, to avoid frequent switching of the sampling strategy near the critical point. In some embodiments, the redundant sensor data fusion can use different architectures such as sensor-level fusion or state vector-level fusion. The sensor-level fusion directly combines the original measurement values, and the state vector-level fusion fuses the state estimates processed independently by each sensor. Optionally, the sliding mode surface of the adaptive sliding mode control algorithm can be designed as a Euclidean distance function from the current position of the instrument to the expected path. Optionally, the application of the path correction amount can use a feedforward plus feedback method, the feedforward part compensates based on the prediction model, and the feedback part corrects based on the current deviation. Optionally, the order of the B-spline curve interpolation can be selected according to the path smoothness requirement, usually using a third-order or fourth-order B-spline. It can be understood that the core of the adaptive navigation adjustment module is to convert the stability evaluation result into specific and executable path control commands.

[0075] In specific implementations, the calculation of the position drift indicator relies on a fixed length sliding time window, which contains the prediction and observation residual sequence of the last N sampling instants, and the real-time position drift indicator value is obtained by calculating the root mean square error of the sequence. In specific implementations, when the redundant sensor data fusion is enabled, the system will dynamically initialize an additional Kalman filter, which runs in a master-slave mode, and its observation input comes from both the main sensor and the redundant sensor. In specific implementations, the control gain adjustment law of the adaptive sliding mode control algorithm is usually represented as a linear or piecewise linear function of the navigation deviation coefficient, and the function relationship is calibrated through pre-off-line control system simulation. In specific implementations, the path correction amount applied to the control point is a geometric transformation process, usually involving translation and rotation operations on the control point coordinates, and the transformation matrix is determined by the correction amount output by the adaptive sliding mode control algorithm. In specific implementations, the coordinate sequence of the optimized navigation instruction is sent to the underlying motion controller of the microsurgical robot in the communication protocol and data structure.

[0076] In specific implementations, considering a specific example, when the surgical instrument is performing fine operations in a narrow lumen, due to the increase in tissue interaction force, the position drift indicator of the instrument may increase significantly from 0.5 mm to 1.2 mm. After the real-time data monitoring module detects this change, it immediately increases the sampling frequency of the optical positioning system from 100 Hz to 250 Hz, and activates the data stream of the backup inertial measurement unit. The attitude data of the main and backup inertial measurement units are Kalman filtered and fused to obtain more reliable instrument attitude estimation. At the same time, the navigation stability evaluation module may calculate a higher navigation deviation coefficient, for example, 0.8 (normalized value), and the adaptive navigation adjustment module calculates a path correction vector pointing to the center of the lumen according to the deviation coefficient using the adaptive sliding mode control algorithm. The correction vector is applied to the control point of the current path, and a more optimized path that is closer to the center line of the lumen and has smoother curvature changes is generated through the B-spline curve reconstruction algorithm, and the coordinate sequence of the path is sent to the robot controller to guide the instrument to safely pass through the narrow area. It can be understood that the implementation of embodiment 5 demonstrates a complete closed-loop adaptive adjustment process from state perception to control execution.

[0077] In specific implementation, the increase of sensor sampling frequency will correspondingly increase the computational load of the central processing unit, and the system design needs to ensure dynamic adjustment within the processing capacity range to avoid real-time decline due to resource overload. In specific implementation, the adaptive sliding mode control algorithm needs to include anti-chattering measures, such as replacing the sign function with a saturation function, to reduce the impact of the inherent high-frequency chattering phenomenon of sliding mode control on the smooth movement of the robot. In specific implementation, the optimized new path needs to be verified by a collision detection algorithm to ensure that it does not interfere with key anatomical structures, and then confirmed as the final optimized navigation instruction. In specific implementation, the execution frequency of the entire adaptive adjustment process is usually lower than the sensor sampling frequency, but needs to meet the real-time requirements of robot control, such as being set to 50 Hz to 100 Hz.

[0078] While embodiments of the present application have been shown and described, it is to be understood that the embodiments described are merely divergences, modifications, replacements and variations of the embodiments, and the scope of the present application is defined by the appended claims and their equivalents.

Claims

1. A surgical navigation system for a microsurgical robot using ultrasound fusion multimodal image guidance, characterized in that, The system includes: The system initialization module receives surgical planning parameters, configures navigation reference values, and generates a system activation signal. The multimodal image acquisition module controls the ultrasound imaging device and complementary imaging device to acquire real-time anatomical data based on the system activation signal, performs multi-source image alignment and quality verification, and outputs a calibrated multimodal image set; The real-time data monitoring module tracks the movement trajectory and posture angle of surgical instruments based on the multimodal image set, analyzes the trend of instrument position change, and generates dynamic tracking data of the instruments. The image fusion analysis module performs spatial transformation and content overlay of ultrasound and multimodal images based on the instrument dynamic tracking data and multimodal image set, evaluates fusion consistency, and generates fused image results and fusion quality parameters. The navigation stability assessment module uses the fused image results, fusion quality parameters, and device dynamic tracking data to calculate the deviation tolerance of the navigation system and outputs a stability assessment report. The adaptive navigation adjustment module adjusts the curve smoothness and fault tolerance range of the navigation path based on the stability assessment report, and generates optimized navigation instructions.

2. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 1, characterized in that, The specific steps for the multimodal image acquisition module to output the calibrated multimodal image set are as follows: The system simultaneously triggers the ultrasound probe and optical coherence tomography scanner to capture the raw image stream, recording the timestamp and equipment parameters for each image frame. Multi-scale filtering is applied to the raw image stream to reduce noise interference, and a contrast-limited adaptive histogram equalization method is used to enhance image details. Edge intensity and texture uniformity features of the enhanced image are extracted, and the image sharpness index is calculated. The image sharpness index is compared with a preset sharpness threshold; if the sharpness index is lower than the threshold, the equipment focusing parameters are automatically adjusted or the image is reacquired. All qualified image frames are integrated and arranged chronologically to generate a multimodal image set.

3. The ultrasound-fused multimodal image-guided surgical navigation system for microsurgical robots as described in claim 2, characterized in that, The specific steps by which the image fusion analysis module generates fused image results and fusion quality parameters are as follows: Scale-invariant feature transformation detection is performed on ultrasound images and multimodal images to generate a set of keypoint descriptors. The keypoint descriptors are matched using a random sample consensus algorithm to estimate the affine transformation matrix between images. The ultrasound images are mapped to the multimodal image coordinate system using the affine transformation matrix, and the pixel alignment error is calculated. The consistency of overlapping areas of the images is analyzed based on the pixel alignment error to generate a fusion quality score. The aligned image pixel values ​​are fused using a weighted average method, and the fused image result is output.

4. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 3, characterized in that, The image fusion analysis module also includes the following steps for evaluating fusion consistency: The process involves acquiring fused image results from continuous time series and constructing an image stability time series; performing empirical mode decomposition on the time series to obtain intrinsic mode function components; calculating the sample entropy value of each component to quantify complexity; analyzing the fluctuation range of the sample entropy value and deriving the image temporal stability index; and generating comprehensive fusion quality parameters by combining pixel alignment error and temporal stability index.

5. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 4, characterized in that, The specific steps for the image fusion analysis module to generate fusion quality parameters are as follows: Pixel alignment error and image temporal stability index are normalized into feature vectors; a decision tree model is used to perform regression analysis on the feature vectors to predict the fusion quality level; and the specific values ​​of the fusion quality parameters are output according to the quality level mapping table.

6. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 5, characterized in that, The specific steps for the navigation stability assessment module to output a stability assessment report are as follows: Receive fusion quality parameters and device dynamic tracking data; set multi-level quality tolerance thresholds to classify fusion quality parameters into high consistency, medium consistency and low consistency levels; for each level, calculate the navigation deviation coefficient in combination with the device position fluctuation; evaluate the system reliability based on the navigation deviation coefficient and generate a stability assessment report.

7. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 6, characterized in that, The specific steps for the real-time data monitoring module to generate dynamic tracking data for the medical device are as follows: The system continuously acquires the instrument's three-dimensional coordinates and Euler angles; applies an extended Kalman filter algorithm to smooth and predict the data, estimating the instrument's true position; calculates the residual sequence between the observed and predicted positions; analyzes the root mean square error of the residual sequence to obtain the position drift index; triggers a calibration procedure when the position drift index exceeds a dynamic threshold, otherwise maintains normal tracking; and outputs dynamic tracking data of the instrument containing the position and drift index.

8. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 7, characterized in that, The real-time data monitoring module also adaptively adjusts the sampling strategy based on the position drift index: when the position drift index increases, the sensor sampling frequency is increased and redundant sensor data fusion is enabled; when the position drift index decreases, the default sampling settings are restored to optimize resource utilization.

9. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 8, characterized in that, The specific steps for the adaptive navigation adjustment module to generate optimized navigation instructions are as follows: The system analyzes the navigation deviation coefficient in the stability assessment report and the position drift index in the instrument dynamic tracking data; it uses an adaptive sliding mode control algorithm to calculate the path correction, where the control gain dynamically changes according to the deviation coefficient; it applies the correction to the current navigation path control point to generate a new path with a smooth transition; and it outputs the coordinate sequence of the new path as an optimized navigation command.

10. The ultrasound fusion multimodal image-guided surgical navigation system for microsurgical robots as described in claim 5, characterized in that, The specific steps for training and applying the decision tree model are as follows: Collect a historical feature vector dataset labeled with the true fusion quality level; use the CART algorithm to generate a decision tree model on the training set, and avoid overfitting through pruning strategies; In the application phase of the decision tree model, the real-time normalized feature vectors are used for condition judgment starting from the root node of the decision tree and traversed to the leaf node. Output the fusion quality level prediction result corresponding to the leaf node, and calculate the probability confidence level of its belonging to that level.

Images