Fusion ai visualized panoramic stitching for neuroendoscopic robotic navigation

Abstract

The application relates to the technical field of surgical robot navigation, and particularly discloses a neuroendoscope robot navigation method fusing AI visualized panorama splicing, which is applied to a surgical robot system comprising a neuroendoscope, a six-degree-of-freedom mechanical arm and a computing unit, and comprises the following steps: acquiring an initial registration transformation matrix of a patient world coordinate system and a mechanical arm base coordinate system and a camera intrinsic parameter matrix. The method constructs a double-track decoupling architecture comprising an asynchronous slow track vision layer and a high-frequency fast track control layer, so that time-consuming image three-dimensional reconstruction and low-delay mechanical arm pose calculation are independently operated, the blockage of control feedback by visual calculation is avoided while the continuous monitoring of the mechanical arm state is maintained. In a specific surgical operation, according to the real-time spatial coordinates of the instrument end, the three-dimensional panorama dissection grid model is dynamically divided into a core operation area, a transition connection area and a global reference area, and a non-rigid deformation energy function is separately solved for the core operation area where the tissue is compressed.

Description

Technical Field

[0001] This invention relates to the field of surgical robot navigation technology, specifically to a neuroendoscopic robot navigation method that integrates AI-powered visual panoramic stitching. Background Technology

[0002] In neuroendoscopic robotic surgery, navigation systems provide positioning information for surgical instruments by spatially registering preoperative three-dimensional medical images with the patient's intraoperative anatomy. Existing navigation methods primarily rely on static preoperative image data and rigid registration algorithms.

[0003] During actual surgery, brain tissue typically undergoes non-rigid deformation due to the insertion and traction of instruments, the removal of lesions, and the loss of cerebrospinal fluid. Static anatomical models cannot be updated accordingly, leading to spatial discrepancies between the anatomical locations displayed on the navigation screen and the actual intraoperative conditions.

[0004] To obtain the intraoperative deformation status, some existing technologies have introduced intraoperative visual images for 3D reconstruction to achieve dynamic updates. However, dense 3D reconstruction of endoscopic video streams involves a large amount of computation. If this is placed in a single synchronous processing flow along with the high-frequency kinematic control of the robotic arm, it will cause the computation process to be blocked, resulting in navigation screen delays or loss of robotic arm control signals.

[0005] Meanwhile, existing methods typically update the entire surgical field of view uniformly when processing deformation calculations, without distinguishing the stress areas where the instruments actually exert their force. Since non-rigid deformations are concentrated in the contact area at the instrument tip, performing indiscriminate calculations across the entire field of view increases the system load, resulting in insufficient accuracy in representing the deformation of locally compressed areas and making it difficult to represent the true spatial contact state between the instruments and soft tissues in the images. Summary of the Invention

[0006] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, the purpose of this invention is to propose a neuroendoscopic robot navigation method that integrates AI-powered visual panoramic stitching, providing operators with an intuitive reference for determining the spatial relationship between instrument positions and anatomical structures.

[0007] To achieve the above objectives, a first aspect of the present invention proposes a neuroendoscopic robot navigation method integrating AI-based visual panoramic stitching, applicable to a surgical robot system comprising a neuroendoscope, a six-degree-of-freedom robotic arm, and a computing unit, comprising:

[0008] Obtain the initial registration transformation matrix between the patient's world coordinate system and the robot arm's base coordinate system, as well as the camera intrinsic parameter matrix;

[0009] Construct a dual-track decoupled navigation architecture, including an asynchronous slow track visual layer and a high-frequency fast track control layer;

[0010] The asynchronous slow-track visual layer processes the neuroendoscopic video stream, generates a three-dimensional panoramic anatomical mesh model of the intracranial cavity through depth estimation and mesh reconstruction, and outputs anatomical semantic labels, visually calculated pose, and global rigid deformation correction matrix.

[0011] The high-frequency fast track control layer collects the state data of the robotic arm, calculates the real-time pose coordinates of the end effector based on the kinematic model, and outputs high-frequency pose data with real-time confidence labels.

[0012] Based on the real-time pose coordinates, the three-dimensional panoramic anatomical mesh model is dynamically divided into a core operation area, a transition area, and a global reference area.

[0013] Local non-rigid incremental correction is initiated for the core operation area, image feature points are extracted and inter-frame matching is performed to construct and solve the non-rigid deformation energy function and generate an incremental deformation displacement field.

[0014] By combining the global rigid deformation correction matrix and the incremental deformation displacement field, the comprehensive deformation coordinates of the mesh vertices are calculated through smoothing weight coefficients. Based on the high-frequency pose data and the visually calculated pose, a projection matrix is ​​constructed to reproject the three-dimensional mesh vertices onto the two-dimensional image plane to generate a fused navigation screen.

[0015] To achieve the above objectives, a second aspect of the present invention proposes a neuroendoscopic robot navigation system integrating AI-based visual panoramic stitching, applicable to a surgical robot system comprising a neuroendoscope, a six-degree-of-freedom robotic arm, and a computing unit, including:

[0016] The parameter acquisition module is used to obtain the initial registration transformation matrix between the patient's world coordinate system and the robot arm's base coordinate system, as well as the camera's intrinsic parameter matrix.

[0017] The dual-track architecture module is used to build a dual-track decoupled navigation architecture, including an asynchronous slow track visual layer and a high-frequency fast track control layer;

[0018] The slow-track vision processing module is used to enable the asynchronous slow-track vision layer to process the neuroendoscopic video stream, generate a three-dimensional panoramic anatomical mesh model of the intracranial cavity through depth estimation and mesh reconstruction, and output anatomical semantic labels, visually calculated pose and global rigid deformation correction matrix.

[0019] The fast track control calculation module is used to enable the high-frequency fast track control layer to collect the state data of the robotic arm, calculate the real-time pose coordinates of the end effector based on the kinematic model, and output high-frequency pose data with real-time confidence labels.

[0020] The dynamic partitioning module is used to dynamically divide the three-dimensional panoramic anatomical mesh model into a core operation area, a transition area, and a global reference area based on the real-time pose coordinates.

[0021] The local correction calculation module is used to initiate local non-rigid incremental correction for the core operation area, extract image feature points and perform inter-frame matching to construct and solve the non-rigid deformation energy function and generate an incremental deformation displacement field.

[0022] The image fusion projection module is used to combine the global rigid deformation correction matrix and the incremental deformation displacement field, calculate the comprehensive deformation coordinates of the mesh vertices through smoothing weight coefficients, and construct a projection matrix based on the high-frequency pose data and the visually calculated pose to reproject the three-dimensional mesh vertices onto the two-dimensional image plane to generate a fused navigation screen.

[0023] To achieve the above objectives, a third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein when the computer program is executed by the processor, it implements the above-described method for fusion AI visualization panoramic stitching for neuroendoscopic robot navigation.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0025] The method of this invention constructs a dual-track decoupled architecture comprising an asynchronous slow-track vision layer and a high-frequency fast-track control layer. This allows the time-consuming 3D image reconstruction and the low-latency robotic arm pose calculation to run independently, maintaining continuous monitoring of the robotic arm's state while avoiding the obstruction of control feedback by visual computation. In specific surgical operations, this scheme dynamically divides the 3D panoramic anatomical mesh model into a core operating area, a transition area, and a global reference area based on the real-time spatial coordinates of the instrument's end effector. For the core operating area where tissue compression occurs, the non-rigid deformation energy function is solved separately.

[0026] This partitioned processing method concentrates the system's computing power on the key anatomical areas of instrument interaction, improving the computational efficiency and display accuracy of local deformation tracking. By integrating deformation data through smoothing weight coefficients and reprojecting it onto a two-dimensional image plane, the navigation system can continuously output images reflecting the deformation state of soft tissue under the action of instruments, providing operators with an intuitive and accurate reference for judging the spatial relationship between instrument position and anatomical structures. Attached Figure Description

[0027] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0028] Figure 1 This is a flowchart illustrating the neuroendoscopic robot navigation method integrating AI visualization and panoramic stitching provided by the present invention.

[0029] Figure 2 This is a nonlinear mapping surface diagram of multi-source physical interference and pose confidence of the robotic arm in the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by the present invention.

[0030] Figure 3 This is a spatial distribution curve of deformation smoothing weights from the core to the global region in the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by this invention.

[0031] Figure 4 This is an iterative convergence curve of the total energy function of local non-rigid deformation in the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by the present invention.

[0032] Figure 5 This is a time series diagram of the number of endoscopic feature points extracted under sudden occlusion conditions in the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by the present invention.

[0033] Figure 6 This is a nonlinear response surface plot of biomechanical contact stress and deformation displacement in the core operating area of ​​the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by the present invention.

[0034] Figure 7 This is a multi-source pose Kalman filter trajectory tracking diagram under high-frequency micro-oscillation in the neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching provided by the present invention;

[0035] Figure 8 This is a schematic diagram illustrating the implementation of the neuroendoscopic robot navigation system with AI-integrated visualization and panoramic stitching provided by the present invention.

[0036] Figure 9 This is a schematic diagram of the electronic device provided by the present invention. Detailed Implementation

[0037] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0038] The following description, with reference to the accompanying drawings, describes a neuroendoscopic robot navigation method, system, and electronic device that integrates AI-based visual panoramic stitching according to embodiments of the present invention.

[0039] Example 1:

[0040] like Figure 1As shown, this embodiment provides a neuroendoscopic robot navigation method that integrates AI visualization and panoramic stitching. This method is applied to a surgical robot system that includes a neuroendoscopy, a six-degree-of-freedom robotic arm, and a computing unit.

[0041] In neurosurgery, neuroendoscopy is used to provide an optical field of view within the brain, a six-degree-of-freedom robotic arm is used to hold and drive neuroendoscopy or surgical instruments for high-precision spatial positioning, and the computing unit includes a graphics processor (GPU) responsible for processing high-concurrency image data and an industrial computer (IPC) responsible for hard real-time logic control.

[0042] The method provided in this embodiment solves the navigation inaccuracy problem caused by intraoperative soft tissue deformation through software algorithm architecture design and multimodal data fusion calculation. To achieve this objective, the method of this embodiment specifically includes the following:

[0043] For example, the method in this embodiment first performs spatial registration and parameter calibration steps. The system needs to obtain the initial registration transformation matrix of the patient's world coordinate system and the robot arm's base coordinate system, as well as the camera intrinsic parameter matrix.

[0044] The patient world coordinate system is a reference coordinate system established based on optical tracking markers fixed to the patient's head. Preoperative MRI or CT 3D images are mapped to this coordinate system. The robotic arm base coordinate system is the underlying physical coordinate system of the robot control system. Preoperative point cloud acquisition is achieved by using calibration probes to collect point cloud data at characteristic sites on the patient's head, and spatial transformation parameters are calculated using singular value decomposition algorithms. This yields the initial registration transformation matrix mapping the patient world coordinate system to the robotic arm base coordinate system. The camera intrinsic parameter matrix reflects the inherent parameter matrix of the neuroendoscopic lens's focal length, principal point coordinates, and distortion coefficients. It is obtained preoperatively through the Zhang Zhengyou calibration method for specific optical lens groups.

[0045] Through the above steps, the system establishes a unified mathematical mapping benchmark between physical space, image space, and optical imaging space.

[0046] Specifically, after initialization is completed, this embodiment constructs a dual-track decoupled navigation architecture, which includes an asynchronous slow track vision layer and a high-frequency fast track control layer.

[0047] In traditional surgical navigation architectures, computationally intensive image rendering and millisecond-level motor servo control often reside in the same main thread or synchronous blocking thread. When large-area bleeding occurs during surgery, causing a sharp increase in the time consumed by the image reconstruction algorithm, it can easily lead to delayed response of robotic arm control commands.

[0048] To avoid this risk, this embodiment decouples visual processing and physical motion control at the operating system level, both physically and logically. The asynchronous slow-track vision layer is deployed in a GPU parallel computing environment with high throughput characteristics, focusing on processing unstructured video stream data, and its runtime is limited by the complexity of image rendering; the high-frequency fast-track control layer is deployed in a computing unit equipped with a hard real-time operating system, using an industrial Ethernet bus protocol (such as EtherCAT) to ensure the latency-free acquisition and processing of the robotic arm's kinematic data with a strict clock synchronization mechanism.

[0049] Optionally, the asynchronous slow-track visual layer processes the neuroendoscopic video stream, generates a three-dimensional panoramic anatomical mesh model of the intracranial cavity through depth estimation and mesh reconstruction, and outputs anatomical semantic labels, visually calculated pose, and a global rigid deformation correction matrix. Because the neuroendoscope moves within the narrow space of the intracranial cavity, its single-frame field of view is limited and lacks depth information.

[0050] In this step, the asynchronous slow-track vision layer performs dense feature extraction and 3D reconstruction on the continuously incoming 2D video stream to generate a 3D panoramic anatomical mesh model covering the entire surgical area.

[0051] Specifically, the asynchronous slow-track visual layer processes the neuroendoscopic video stream, including multi-scale partitioning preprocessing of single-frame endoscopic images.

[0052] Since the central region of the endoscopic field of view is the primary operating area, and the peripheral regions often suffer from severe optical distortion, the preprocessing algorithm divides the image into a high-resolution central scale region and a low-resolution peripheral scale region. Subsequently, a depth estimation network outputs a pixel-level dense depth map and the aforementioned anatomical semantic labels. The depth estimation network employs a convolutional neural network architecture trained on a clinical anatomy dataset. Its input is the multi-scale preprocessed two-dimensional image, and its output is a depth scalar value corresponding to each pixel, as well as an anatomical semantic label reflecting the current tissue type. The anatomical semantic label classifies different pixel regions into categories such as blood vessels, nerves, ventricular walls, or tumor entities.

[0053] It is also important to note that during this process, the system acquires a preoperative reference point cloud, generates an intraoperative reconstructed point cloud based on the extracted feature point 3D coordinates, and uses a rigid registration algorithm to minimize the sum of squared Euclidean distances between the preoperative reference point cloud and the intraoperative reconstructed point cloud, thus obtaining the global rigid deformation correction matrix. The preoperative reference point cloud is generated from the tissue surface contour extracted by preoperative image segmentation; the intraoperative reconstructed point cloud is a 3D point set generated based on depth map transformation. The algorithm constructs an iterative nearest-point objective function to calculate the optimal rotation and translation parameters that maximize the spatial overlap between the preoperative and intraoperative point clouds.

[0054] For example, the mathematical model of its objective function is expressed as follows:

[0055] ;

[0056] in, The objective function representing the sum of squared Euclidean distances for rigid registration; This represents the total number of valid 3D matching point pairs extracted; This represents the global rotation matrix to be solved; The preoperative baseline point in the cloud represents the first The three-dimensional coordinate vector of each feature point; This represents the global translation vector to be solved; Indicating the first point in the intraoperative reconstruction point cloud The three-dimensional coordinate vector of each feature point.

[0057] The solution obtained and Together, they constitute the global rigid deformation correction matrix, which is used to compensate for the overall displacement of brain tissue caused by cerebrospinal fluid release.

[0058] Optionally, the high-frequency fast track control layer acquires the robotic arm's state data, calculates the real-time pose coordinates of the instrument's end effector based on a kinematic model, and outputs high-frequency pose data with real-time confidence labels. Each joint of the six-DOF robotic arm is equipped with a high-precision absolute encoder, and the high-frequency fast track control layer acquires the angle feedback values ​​of these encoders at kilohertz-level frequencies. Based on a pre-established Denavit-Hartenberg (DH) link parametric forward kinematics model, the system can convert the angle values ​​of each joint into real-time spatial coordinates of the surgical instrument's end effector in the robotic arm's base coordinate system.

[0059] Specifically, due to the complex physical interference within the operating room, the high-frequency fast track control layer outputs high-frequency pose data with real-time confidence labels. This requires real-time acquisition of four types of interference data: joint transmission gap fluctuation, motor temperature drift, electromagnetic interference intensity, and physiological displacement amplitude. After prolonged operation, the robotic arm's harmonic reducer experiences minor mechanical wear, manifesting as joint transmission gap fluctuation. The heat generated by the joint motor during prolonged operation causes thermal expansion and contraction of the transmission components, resulting in motor temperature drift. The operation of high-frequency equipment such as surgical electrosurgical units generates electromagnetic interference. The slight head tremors caused by the patient's breathing or heartbeat manifest as physiological displacement amplitude.

[0060] The system quotients the measured values ​​of various types of interference data with their corresponding critical thresholds, and then performs a weighted summation using weighting factors to obtain the real-time interference deviation value. This calculation process is implemented through the following formula:

[0061] ;

[0062] in, This represents the calculated real-time interference deviation value; , , , These represent the weighting factors corresponding to the four types of interference sources determined experimentally, and their sum is 1; This represents the real-time measured value of the joint transmission clearance fluctuation; This represents the critical threshold for the maximum permissible fluctuation of joint transmission clearance in a mechanical system. This represents the measured value of the motor temperature drift; This indicates the critical threshold for temperature drift that enables safe operation of the motor. This represents the measured value of the electromagnetic interference intensity collected by the sensor. This represents the critical threshold for the system's resistance to electromagnetic interference. This represents the measured value of the patient's physiological displacement amplitude; This represents the critical threshold for physiological displacement amplitude within the tolerance range.

[0063] Subsequently, based on the interval distribution characteristics of the real-time interference deviation values, they are discretely mapped to real-time confidence labels of different numerical levels. When the interference deviation value is at an extremely low level, the pose calculated by the system is determined to have extremely high confidence, and it is mapped to the highest level of real-time confidence label; as the interference deviation value increases, the label level is gradually reduced, thereby quantifying the impact of the underlying physical interference on the top-level pose accuracy.

[0064] like Figure 2 This diagram illustrates the nonlinear mapping relationship between multi-source physical disturbances to the robotic arm and the output pose confidence level. The horizontal axis represents the ratio of mechanical system deviations, and the horizontal axis represents the ratio of external environmental disturbances; both are dimensionless. The vertical axis represents the level of the real-time pose confidence label.

[0065] The surface in the figure uses different colors to distinguish the distribution of confidence values. The bright, flat areas with warm colors correspond to the low interference state where the interference ratio is between 0 and 0.8. At this time, the surface is stably maintained at a high confidence level of 1, ensuring that the system is in a highly reliable operating range.

[0066] As the ratio of mechanical deviation to environmental disturbance continues to increase in the range of 1.5 to 2, the surface exhibits a step-like non-linear decline, passing through a medium confidence plateau of 0.6, and finally falling to a low confidence trough of 0.3, corresponding to the cool-toned part in the figure.

[0067] During the transition interval of confidence level switching, the surface exhibits a continuous and steep descent pattern. This waveform transformation characteristic indicates that the system's confidence evaluation algorithm has set up a smoothing buffer mechanism when processing discrete level mapping, which suppresses frequent jumps in the critical state.

[0068] Meanwhile, in the extremely high interference region where the interference ratio is greater than 1.5, the surface data exhibits obvious burr ripples and high-frequency oscillation curves. This graphical representation truly reflects the decrease in the signal-to-noise ratio of interference deviation calculation when the current underlying sensor data faces strong electromagnetic coupling and mechanical extreme wear, resulting in a certain degree of numerical dispersion and delay in the mapped output. This indicates that there is still room for further research and development to improve the anti-interference convergence speed and edge stability of the multi-source fusion filtering algorithm under extreme conditions.

[0069] For example, due to the non-rigid deformation of brain tissue, such as the indentation caused by surgical forceps pressing on brain tissue, the indentation is mainly concentrated in the limited area directly in contact with the instrument, while brain tissue farther away from the instrument remains basically rigid. To optimize the allocation of computational resources, this embodiment dynamically divides the three-dimensional panoramic anatomical mesh model into a core operating area, a transition area, and a global reference area based on the real-time pose coordinates.

[0070] Specifically, the step of dynamically dividing the three-dimensional panoramic anatomical mesh model into a core operation area, a transition area, and a global reference area includes: the system calculating in real time the Euclidean distance from any mesh vertex in the three-dimensional panoramic anatomical mesh model to the end of the instrument.

[0071] If the Euclidean distance is less than or equal to a first distance threshold, the corresponding grid vertex is assigned to the core operating area. The core operating area represents the region where the instrument is directly acting on the tissue during physical cutting, traction, or electrocoagulation; tissue deformation within this region is the most intense and irregular.

[0072] If the Euclidean distance is greater than the first distance threshold and less than or equal to the second distance threshold, the corresponding mesh vertex is assigned to the transition zone. The transition zone represents a stress diffusion area that undergoes slight deformation due to force transmission.

[0073] If the Euclidean distance is greater than the second distance threshold, the corresponding mesh vertex is assigned to the global reference region. The global reference region represents the area that is not directly affected by external forces from the device, but only by gravity or cerebrospinal fluid loss, resulting in slow overall displacement.

[0074] Wherein, the second distance threshold is greater than the first distance threshold. As a configuration setting in engineering implementation, the first distance threshold can be set to 5 mm, and the second distance threshold can be set to 15 mm.

[0075] Simultaneously, the anatomical semantic labels are used to forcibly classify high-risk anatomical structures into the core operating area. For example, when the anatomical semantic labels output by the depth estimation network identify vital high-risk structures such as the internal carotid artery or optic nerve, even if they are more than a first distance threshold from the instrument tip, the system will forcibly classify them into the core operating area through logical intervention to ensure the highest level of monitoring of even minor deformations of these high-risk structures.

[0076] Optionally, local non-rigid incremental correction is initiated for the core operating area, image feature points are extracted and inter-frame matching is performed to construct and solve the non-rigid deformation energy function, generating an incremental deformation displacement field. This step aims to accurately capture tissue surface deformation caused by instrument pressure through the microscopic flow of visual features.

[0077] Specifically, the step of extracting image feature points and performing inter-frame matching to construct and solve the non-rigid deformation energy function and generate an incremental deformation displacement field includes: extracting sparse feature points within the core operating region and performing inter-frame matching, and obtaining a set of valid three-dimensional feature point pairs after eliminating mismatches. For consecutive video frames, the system applies the Accelerated Robust Feature Algorithm (SURF) or deep learning-based feature descriptors to extract key pixels within the core operating region, and assigns them three-dimensional spatial coordinates based on the aforementioned generated dense depth map. Inter-frame matching is completed by determining the Euclidean distance ratio of the feature vectors, and the Random Sample Consensus Algorithm (RANSAC) is used to eliminate incorrectly matched point pairs caused by reflections or occlusions.

[0078] The system constructs a total energy function comprising a data fitting error term and a smoothing constraint term. The data fitting error term constrains the coordinate deviation of matching feature points before and after deformation, while the smoothing constraint term is constructed based on the normalized Laplacian matrix of the 3D panoramic anatomical mesh model. The mathematical relationship of its total energy function is expressed as follows:

[0079] ;

[0080] in, This represents the total energy function to be optimized; This represents the energy value of the data fitting error term; Regularization weights that control the degree of smoothness; This represents the energy value of the smoothing constraint term.

[0081] The above data fitting error term is specifically expressed as follows:

[0082] ;

[0083] in, This indicates the number of valid 3D feature point pairs acquired within the core operating area; Indicates the current moment in the video frame. Three-dimensional observation coordinates of one effective feature point; This represents the three-dimensional reference coordinates of the feature point corresponding to the previous time step; The term to be solved is the first... The incremental deformation displacement vector of each feature point from the previous time step to the current time step.

[0084] By constraining the consistency between the observed coordinates and the deformed coordinates, the solution of the displacement field is ensured to closely match the actual visual observation data. Simultaneously, a smoothing constraint term based on the Laplacian operator of the mesh vertices is added. This ensures that biological soft tissues will not experience physically abnormal tearing or sharp mutations during deformation.

[0085] Finally, the deformation parameters that minimize the total energy function are solved using the Gauss-Newton method or the nonlinear conjugate gradient method, yielding the incremental deformation displacement field of the mesh vertices within the core operating region. This displacement field contains the spatial movement direction and distance of each micro-mesh vertex within the core operating region.

[0086] like Figure 4 The iterative convergence process of solving the total energy function when initiating local non-rigid incremental correction for the core operating region is illustrated. In the figure, the horizontal axis represents the number of iterations, and the vertical axis represents the residual value of the energy function, in dimensionless units.

[0087] In the figure, the solid black line represents the total energy function curve, the dashed blue line represents the data fitting error term curve, and the dotted red line represents the smoothing constraint term curve.

[0088] Observing the trend of the graph, it can be seen that in the first 15 iterations, the curves of the data fitting error term and the total energy function exhibit a steep downward waveform, with the residual value of the total energy function rapidly decreasing from around 105 to about 20. This indicates that the system has a high initial response speed for solving the nonlinear gradient of the micro-texture displacement of the tissue surface, which can meet the computational power requirements of high-frequency dynamic updates in surgical navigation.

[0089] Meanwhile, the curve of the smoothing constraint term represented by the red dotted line shows a reasonable increase in value in the early stage of iteration as the three-dimensional anatomical mesh deforms, and tends to stabilize around the value of 12. This indicates that the mesh smoothing mechanism based on the normalized Laplacian matrix plays a role and effectively prevents excessive distortion and topological tearing of the anatomical model during the calculation process.

[0090] However, careful observation of the blue dashed line's shape during the 15th to 25th iterations reveals that due to potential minute reflective noise or dynamic blood flow interference within the core operating area, the data fitting error term curve exhibits slight oscillations in this interval. This prevents the total energy residual from ultimately decreasing to zero, instead converging and stabilizing at a noise floor of approximately 17. This fluctuation and non-zero convergence objectively reflect the limitations of visual inter-frame matching in complex clinical optical environments, indicating that there is still room for further improvement in the system's underlying effective feature point filtering and dynamic interference-resistant energy function optimization algorithms.

[0091] For example, by combining the global rigid deformation correction matrix and the incremental deformation displacement field, the comprehensive deformation coordinates of the mesh vertices are calculated through smoothing weight coefficients, and a projection matrix is ​​constructed based on the high-frequency pose data and the visually calculated pose, so as to reproject the three-dimensional mesh vertices onto the two-dimensional image plane to generate a fused navigation screen.

[0092] Specifically, to avoid mesh model tearing at the boundaries of different dynamic partitions, the step of calculating the comprehensive deformation coordinates of mesh vertices using smoothing weight coefficients and constructing a projection matrix based on the high-frequency pose data and the visually calculated pose includes: assigning fixed smoothing weight coefficients to mesh vertices in the core operating region and the global reference region, respectively. The weight coefficient for the core operating region is fixed to a value representing full acceptance of incremental deformation correction, while the weight coefficient for the global reference region is fixed to a zero-value representing acceptance of only global rigid deformation.

[0093] For the mesh vertices in the transition zone, a cosine smoothing transition function is used to calculate their dynamic smoothing weight coefficients. The smoothing properties of the cosine function result in a natural stress attenuation effect at the mesh vertices from the region of severe deformation to the region of rigid stability. Its function is defined as:

[0094] ;

[0095] in, This represents the smoothing weight coefficient for a specific mesh vertex within the transition zone, with its value smoothly and monotonically transitioning between 0 and 1. This represents the real-time Euclidean distance from the vertex of the grid to the end of the instrument; This represents the first distance threshold in the aforementioned dynamic partitioning; This represents the second distance threshold in the aforementioned dynamic partitioning. Using this weighting coefficient to interpolate and scale the incremental deformation displacement field ensures the topological continuity of the overall anatomical mesh model.

[0096] like Figure 3The graph shows the spatial distribution curves of deformation smoothing weights from the core to the global region. The horizontal axis represents the spatial distance from the instrument's end point in millimeters, and the vertical axis represents the smoothing transition weight coefficients in dimensionless units.

[0097] The red solid line in the figure represents the theoretical cosine decay curve of the system design, and the dark blue scatter dots represent the measured weight distribution of the discrete grid vertices calculated in real time during the operation.

[0098] Observing the trend of the graph, it can be seen that in the core operating area where the spatial distance from the instrument tip is less than or equal to 5 mm, the theoretical curve and the scatter points are maintained at the level of 1, indicating that all the anatomical grid vertices in this area have received deformation data with local incremental correction.

[0099] When the spatial distance is in the transition zone of 5 mm to 10 mm, the theoretical curve exhibits a continuously decreasing cosine waveform transformation, causing the weighting coefficient to smoothly decay from a value of 1 to a value of 0. The dark blue scatter points in the corresponding region follow the distribution of the theoretical curve, ensuring a smooth transition between mesh models at the boundaries of different dynamic partitions. When the spatial distance exceeds 10 mm, it enters the global reference region, where the weighting coefficient drops to a value of 0, indicating that only global rigid deformation compensation is retained in this region.

[0100] Meanwhile, by observing the subtle distribution characteristics of the dark blue scatter points, it can be found that in a real surgical computing environment, limited by the physical resolution of the visual sensor's depth estimation and the floating-point truncation effect in the underlying parallel computing process, the measured grid vertex weights exhibit a certain degree of numerical divergence and high-frequency fluctuation noise above and below the theoretical curve. This scatter point deviation objectively reflects the engineering accuracy limit of the current system when processing massive high-concurrency 3D grid data, indicating that the system still has room for further development in terms of computing power scheduling optimization and noise reduction filtering mechanisms.

[0101] After calculating the comprehensive deformation coordinates, the system needs to map the 3D spatial model back to the 2D screen observed by the surgeon. The system combines the camera intrinsic parameter matrix and the initial registration transformation matrix, using the real-time confidence label as adjustment weights, to perform a weighted fusion calculation on the high-frequency pose data and the visually calculated pose, generating the projection matrix. The visually calculated pose mainly relies on the SLAM algorithm to calculate the endoscope pose through image feature flow; it has high accuracy but is susceptible to occlusion failure. High-frequency pose data relies on mechanical kinematics, which is stable but contains mechanical errors. The mathematical logic of its weighted fusion is as follows:

[0102] ;

[0103] in, This represents the generated projection matrix used for the 3D-to-2D reprojection transformation; This represents the camera intrinsic parameter matrix obtained during the system initialization phase; This represents the real-time confidence label value mapped to a continuous interval of 0 and 1; This represents the high-frequency pose data output by the high-frequency fast track control layer; This represents the visual solution pose output by the asynchronous slow-track visual layer.

[0104] This mechanism ensures that when the robotic arm's transmission is subjected to strong disturbances (reduced confidence), the system automatically increases the dependency weight of the visually calculated pose, and vice versa.

[0105] It is also important to note that surgical robot systems have extremely high safety requirements. The method described in this embodiment further includes a triple cross-validation step based on both global and local dimensions. Specifically, it involves using the high-frequency pose data, the visually calculated pose, and the real-time confidence labels as independent data sources to obtain the global pose deviation by calculating the matrix norm. Specifically, the system compares the transformation matrix obtained from the high-frequency kinematics forward solution with the transformation matrix estimated by visual SLAM, and calculates the Frobenius norm of the difference between the two matrices. This value represents the dispersion of the system's deviation in global physical space localization.

[0106] Subsequently, the local deformation deviation is obtained by calculating the Euclidean distance between the coordinates of the instrument end-effector obtained from visual reconstruction and the coordinates of the physical instrument end-effector. This deviation directly reflects whether the deformation calculation algorithm has diverged or mistracked feature points within the core operating area.

[0107] Finally, based on the magnitude of the global pose deviation and the local deformation deviation, a graded safety response mechanism of the corresponding level is triggered.

[0108] Specifically, the triggering of the corresponding level of graded safety response mechanism includes: when both the global pose deviation and the local deformation deviation are less than or equal to the warning threshold, the system operates normally with all functions, and the navigation algorithm operates normally according to the set logic.

[0109] When the global pose deviation or the local deformation deviation exceeds the warning threshold but is less than the braking threshold, the system enters a warning state. At this point, physical control and visual tracking begin to diverge significantly. The system control navigation interface outputs a corresponding level of alarm prompt (such as a flashing yellow border) and adaptively limits the maximum movement speed of the robotic arm (e.g., limiting the maximum angular velocity of each joint to 20% of its original value). Simultaneously, it forcibly increases the local deformation correction frequency of the core operating area to accelerate the reduction of deviation.

[0110] When the global pose deviation or the local deformation deviation is greater than or equal to the braking threshold, the system is determined to enter a braking state. This state indicates that the system is at risk of damaging non-target brain tissue, and the system needs to intervene immediately. The system immediately triggers the flexible braking of the robotic arm, cuts off the torque output through the underlying servo driver, and engages the motor brake to lock the current position of the endoscope, terminating any automated or semi-automated spatial movement.

[0111] The braking threshold is greater than the warning threshold. The specific physical dimension of both is millimeters, and they need to be determined by the doctor before the operation based on the risk level of the specific surgery.

[0112] Optionally, in order to further optimize the allocation efficiency of GPU computing resources, in the step of initiating local non-rigid incremental correction for the core operating area, the local deformation update frequency is positively correlated with the real-time confidence label.

[0113] When the real-time confidence level is the first confidence level (indicating minimal mechanical and environmental interference), local deformation updates are performed at a first preset frequency (e.g., 15 times per second). At this time, the high-frequency correction can greatly improve the real-time responsiveness of the visual display.

[0114] When the real-time confidence level label is the second or third confidence level (indicating strong electromagnetic interference or vibration interference, and high-frequency feature extraction is prone to introducing noise), the frequency is adjusted to the second or third preset frequency (e.g., reduced to 10 times per second and 5 times per second respectively) to perform local deformation updates, thereby sacrificing a certain display frame rate in exchange for numerical stability of deformation calculation.

[0115] Wherein, the first confidence level is greater than the second confidence level, and the second confidence level is greater than the third confidence level; the first preset frequency is greater than the second preset frequency, and the second preset frequency is greater than the third preset frequency.

[0116] For example, the computing unit ultimately renders and presents the processed data to the surgeon. The reprojection onto the two-dimensional image plane to generate the fused navigation screen employs a hierarchical multi-layer fusion rendering mechanism, including: a three-dimensional panoramic anatomical underlayer layer, a real-time high-definition semi-transparent endoscopic overlay layer, a navigation accuracy heatmap layer, and a hierarchical warning prompt layer, superimposed sequentially from bottom to top. This multi-layer overlay technique avoids the lack of depth perception caused by monotonous wireframe displays.

[0117] The navigation accuracy heatmap maps and labels the three regions of the dynamic partition with different accuracy levels. The core operating area, having undergone high-frequency non-rigid correction, has the closest anatomical position to the actual physical state and is labeled as the first accuracy level, typically rendered as a green heatmap grid. The transition area, due to attenuation caused by interpolation algorithms, has its accuracy somewhat affected and is labeled as the second accuracy level, rendered as a yellow heatmap grid. The global reference area only undergoes rigid compensation and does not calculate local non-rigid deformation; it is labeled as the third accuracy level and rendered as a gray or transparent grid to indicate to the doctor that the displayed position of this area may deviate from the actual tissue.

[0118] In summary, the technical effects of this embodiment can be seen from the existing technology: In the field of computer-aided surgical navigation, traditional technical solutions mostly use static preoperative image data for rigid registration, or attempt to perform global non-rigid deformation calculations in a single computing thread. Because neurosurgery has extremely low tolerance for delay and accuracy, existing technologies either fail to reflect the actual collapse and deformation of brain tissue during surgery, or cause severe system lag and failure due to the highly complex global finite element calculations.

[0119] This embodiment proposes a dual-track decoupled architecture and implements distance-based dynamic partitioning and energy function correction at the software algorithm level. By generating confidence scores through real-time monitoring of robotic arm state disturbances, the system can intelligently determine when to trust visual features and when to trust mechanical transmission. This multi-dimensional integration of AI depth estimation, Laplace smoothing constraints, rigid and non-rigid fusion calculations, and a hard real-time safety verification system not only enables the navigation screen to realistically reflect the indentation and rebound deformation of tissue at the instrument contact point, but also significantly reduces the unnecessary waste of computing resources and effectively avoids the risk of medical accidents caused by algorithm crashes or tracking divergence. It provides a system-level solution for minimally invasive neuroendoscopic robotic surgery that combines high precision, low latency, and high safety.

[0120] Example 2:

[0121] In real neurosurgical endoscopic minimally invasive surgery, the optical field of view of the endoscopic camera is easily affected by transient interference from the external environment, such as blood splatter caused by the rupture of tiny blood vessels obscuring the lens, smoke generated by high-frequency electrosurgical cutting of tissue, or the metal reflection and physical structure of the surgical instruments obscuring the target tissue.

[0122] Under such conditions, the system will face the dilemma of being unable to extract effective physical features from the images due to severe degradation or even complete loss of image information acquired by the endoscope camera.

[0123] Therefore, the core objective of this embodiment is to solve the problem of stagnation or distortion of the three-dimensional navigation screen caused by visual data stream discontinuity by introducing a biomechanical blind spot compensation mechanism based on the mapping of anatomical semantics and force perception.

[0124] To achieve this technical solution, this embodiment will describe in detail a method, which also includes a biomechanical deformation compensation step triggered by sudden intraoperative visual occlusion. This step specifically includes the following:

[0125] Specifically, the system performs monitoring of the preoperative and intraoperative status, continuously monitoring the number of effective sparse feature points extracted by the asynchronous slow-track vision layer within the core operating area.

[0126] In the underlying image processing pipeline of the asynchronous slow-track vision layer, the feature point extraction algorithm performs traversal operations on the pixel gradient matrix of the current frame within each input cycle of the video frame in order to find corner or spot features with spatial invariance.

[0127] Under normal, clear visual conditions, the algorithm can reliably extract hundreds of high-quality feature points from the surface of brain tissue. However, when bleeding occurs in the surgical area, causing the visual field to be covered with red blood, the local contrast and texture details of the image decrease sharply, making it impossible for the feature point extraction algorithm to find pixel coordinates that meet the gradient threshold requirements. The system internally employs a state machine to perform quantity statistics and quality verification on the feature point set output for each frame.

[0128] When the number of effective sparse feature points falls below a preset visual failure threshold, the system is determined to have entered a visual blind zone and the calculation of the non-rigid deformation energy function is suspended. The preset visual failure threshold is a discrete value set based on clinical trial data, representing the minimum data fitting constraint required to maintain the solution of the non-rigid deformation energy function.

[0129] like Figure 5 This figure shows a time series plot of the number of endoscopic feature points extracted under sudden occlusion conditions. The horizontal axis represents time in seconds, and the vertical axis represents the number of effective feature points in units.

[0130] In the figure, the blue solid line represents the real-time feature point count curve, and the red dashed line represents the system's preset visual failure threshold baseline, which is set to 80 points based on experiments.

[0131] Observing the waveform transformation, it can be found that under normal clear vision 20 seconds before surgery, the number of effective feature points is stable at a relatively high level of about 350, which ensures the stable solution of the non-rigid deformation energy function of the core operating area.

[0132] At the 20-second mark, a simulated severe situation of sudden bleeding obstructing the camera lens was encountered. The blue solid line showed a sharp downward trend, falling below the red visual failure threshold baseline in a very short time, and remaining in an extremely low range of around 20 for the next ten seconds. At this point, the system detected the data anomaly, accurately determined that it had entered a visual blind zone, and suspended the calculation, thus cutting off the unreliable visual input source.

[0133] After clearing the field of view in the 40th second, the number of feature points quickly rose back to above the safety threshold. After the system recognized the feature points, it exited the blind zone state and started a smooth transition of multimodal data.

[0134] However, careful observation of the waveform around the 28th second reveals a brief upward spike in the blue solid line during the extremely low-value blind zone, with the value briefly approaching 70. This abnormal spike reflects the engineering limitation of the underlying feature extraction algorithm misclassifying the momentary specular reflection from the endoscopic light source on the surface of fresh blood as a valid physical feature point. This indicates that the system still has room for further upgrades and optimizations in filtering complex optical noise and identifying anti-reflection interference in extremely low-quality images.

[0135] If the energy function is solved by forcibly using a very small number of feature points below the threshold or even mismatched feature points, the solution process will fall into an underdetermined equation state due to the severe lack of observation data. This will lead to a violent and irregular divergence of the output deformation displacement field, resulting in abnormal deformation of the soft tissue model in the navigation screen.

[0136] Therefore, pausing the solution calculation is a necessary intervention to prevent erroneous data from contaminating the 3D panoramic anatomical mesh model. This action cuts off unreliable visual input sources, putting the system into a protective isolation state that awaits multimodal data takeover.

[0137] Optionally, after triggering the isolation state, the system activates the data compensation channel at the physical perception level. In the visual blind zone state, the contact force vector and kinematic penetration depth of the instrument's end effector, collected in real time by the high-frequency fast track control layer, are acquired.

[0138] A high-frequency six-axis torque sensor is integrated into the end-effector flange of the six-degree-of-freedom robotic arm or into the clamping linkage mechanism of the surgical instrument in a neuroendoscopic robotic system. The high-frequency fast track control layer reads the raw strain gauge electrical signal of the sensor at an extremely high sampling frequency via an Ethernet bus, and decouples it into a three-dimensional force vector and a three-dimensional torque vector in Cartesian coordinates through a calibration matrix, thereby obtaining the precise contact force vector when the instrument end-effector physically interacts with the brain tissue.

[0139] Meanwhile, based on the angle feedback of the absolute encoders of each joint of the robotic arm, combined with the kinematic forward model, the system can accurately calculate the three-dimensional spatial coordinates of the current instrument tip, and perform a differential projection calculation on the surface grid coordinates recorded at the moment of initial contact with the tissue to obtain the geometric distance of the instrument tip penetrating or compressing the tissue along the normal vector direction of the tissue surface, that is, the kinematic penetration depth.

[0140] After obtaining these two physical parameters, the system matches and obtains the biomechanical constitutive parameters of the corresponding soft tissue based on the anatomical semantic labels of the grid region where the instrument tip is located. The anatomical semantic labels are output by the depth estimation and semantic segmentation network in the previous step, which assign specific tissue attribute identifiers to each vertex in the three-dimensional panoramic anatomical mesh model, such as gray matter, white matter, meninges, or tumor lesions.

[0141] Because different types of biological tissues exhibit significant anisotropy and differences in hardness and softness in their macroscopic physical and mechanical properties, the system pre-constructs a lookup table of biomechanical properties based on tensile and compressive experiments on ex vivo tissues. By reading the semantic identifier of the current instrument's action grid region and performing key-value matching in the lookup table, the system extracts parameters such as Young's modulus and Poisson's ratio of that specific tissue under its current physiological state; these are collectively referred to as biomechanical constitutive parameters. The introduction of these parameters allows the system to identify whether the instrument is currently pressing on the extremely soft cerebral cortex or the relatively tough dura mater, providing the necessary material mechanics boundary conditions for subsequent deformation calculations.

[0142] For example, after acquiring the aforementioned multi-source cross-modal data, the system enters the core mechanical transformation and calculation engine process.

[0143] Based on the obtained contact force vector, kinematic penetration depth, and biomechanical constitutive parameters, the stress deformation state of the tissue in the contact area is deduced through a biomechanical model, generating a mechanically predicted incremental deformation displacement field.

[0144] In this step, since visual data is interrupted, the system cannot obtain surface deformation through image comparison and instead relies entirely on pure mathematical deduction based on the laws of physics. To meet the hard real-time computing requirements of the surgical navigation system, the system avoids using a computationally time-consuming whole-brain three-dimensional finite element global iterative model and instead adopts a local linear elastic semi-infinite space force deduction model for the core operating area as the biomechanical model.

[0145] This model assumes that within an extremely short time window and a small incision depth, the stress and strain of biological soft tissue exhibit an approximately linear relationship, and that the deformation effect decreases exponentially with increasing distance from the contact center. The GPU parallel computing core within the computing unit treats all mesh vertices within the core operating area as discrete computing nodes to perform force field diffusion calculations. The specific calculation of the incremental deformation displacement field generated by the derivation is derived using the following mathematical formula:

[0146] ;

[0147] In the formula, Representing the Mechanical prediction incremental deformation displacement vector of each grid vertex; The biomechanical constitutive parameters obtained through matching; The composite distribution matrix representing the Poisson's ratio and geometry of the material is used to characterize the volumetric deformation coupling relationship of the material in three orthogonal directions. This represents the obtained contact force vector; The stress attenuation coefficient represents a specific structure and is used to limit the spread of stress deformation. Representing the The Euclidean distance from each grid vertex to the center of force corresponding to the kinematic penetration depth; This represents the obtained kinematic depth of cut; This represents the effective contact area between the device tip and brain tissue, used to convert the contact force vector into contact stress to ensure dimensional consistency in spatial displacement extrapolation.

[0148] By applying the above derivation formula to each grid vertex within the core operating area, the system constructs a vector field entirely based on physical feedback data, namely, a mechanically predicted incremental deformation displacement field. This displacement field characterizes the geometric indentation morphology that should occur in the surrounding brain tissue due to the pressure of the device, even when it is not visible.

[0149] like Figure 6 The graph shows the nonlinear response surface plot of biomechanical contact stress and deformation displacement in the core operating area. The horizontal axis represents the contact stress in kilopascals, the horizontal vertical axis represents the spatial distance from the stress center in millimeters, and the vertical axis represents the deformation displacement in millimeters.

[0150] The three-dimensional surface in the figure uses a color mapping from cool to warm tones to visually reflect the depth distribution of soft tissue collapse. Observing the waveform transformation of the surface in the horizontal axis direction, it can be found that when the spatial distance from the center of force is 0 mm, as the contact stress gradually increases from 0 to 20 kPa, the deformation displacement exhibits a nonlinear characteristic curve that first increases rapidly and then the rate of increase gradually slows down, with the maximum displacement reaching about 3.5 mm. This curve trend objectively reflects that the biomechanical constitutive parameters obtained by the system matching successfully simulate the strain hardening physical properties exhibited by real brain tissue under pressure.

[0151] Observed along the horizontal longitudinal axis, as the spatial distance from the center of force extends from 0 to 5 mm from the boundary of the core operating area, the deformation displacement exhibits a significant spatial exponential decay, gradually transitioning to a blue cool-toned area close to 0 at the edge. This verifies the effectiveness of the local force simulation model in constraining the deformation diffusion range and ensures the feasibility of the blind zone compensation mechanism.

[0152] However, in the extremely deep red stress region of the curved surface where the contact stress exceeds 15 kPa and the spatial distance is close to 0 mm, the surface data exhibits visible fine ripple oscillations and discrete burr patterns. This local high-frequency fluctuation characteristic truly reflects the limitations of the current underlying computing unit in handling local compression at the extreme depth of soft tissue. These limitations are due to the dual constraints of the three-dimensional mesh discretization resolution and the hard real-time solution cycle, resulting in a certain degree of numerical drift and computational truncation error in the output predicted displacement field. This indicates that the numerical integral stability of this biomechanical blind zone compensation system still has room for further upgrades and optimization in dealing with ultra-high load and extreme compression conditions.

[0153] It is important to note that after calculating the physical compensation data, the system must seamlessly integrate it into the existing display pipeline.

[0154] The system replaces the incremental deformation displacement field with the mechanically predicted incremental deformation displacement field and inputs it into the calculation stream of the mesh vertex comprehensive deformation coordinates, thus maintaining the dynamic deformation update of the navigation screen in the core operation area even without visual feedback.

[0155] In the underlying software architecture design, the module for calculating the comprehensive deformation coordinates of mesh vertices is equipped with a multiplexer at its front end. When the state machine is in a visual blind spot, the multiplexer blocks the data link from the visual inter-frame matching solution module while simultaneously enabling the data link from the biomechanical model deduction module. This means that the graphics engine responsible for rendering the 3D panoramic anatomical mesh model no longer receives vertex update coordinates from image feature point tracking, but is directly driven by force and displacement feedback from the robotic arm's end effector.

[0156] In practical applications, this mechanism is crucial: when the surgeon can only see blurry bloodstains in the endoscopic field due to massive bleeding, their instruments may still be performing hemostatic compression. Thanks to this data substitution step, the 3D anatomical model on the navigation screen does not remain static due to visual loss, but rather dynamically displays the gradual compression and indentation of tissue as the surgeon applies pressure and depth with the instruments. This force-positional visualization mapping under blind operation conditions greatly enhances the surgeon's spatial perception and avoids over-puncture or secondary injuries caused by a lack of feedback.

[0157] Specifically, the emergency is usually brief. Once the suction device clears the accumulated blood or the fumes are removed by the flushing fluid, the endoscopic camera will recapture clear images of the local anatomical structures. The system also continuously monitors the feature status, and when the number of effective sparse feature points recovers to above the visual failure threshold, the system determines that it has exited the visual blind zone state.

[0158] After receiving the recovery signal, the state machine will not directly switch the data source back to the vision layer. This is because during the few seconds or even tens of seconds of vision loss, pure biomechanical deduction will inevitably produce a certain numerical integration error, and there is a spatial residual between the theoretical displacement field of the physical deduction and the actual visual displacement field.

[0159] If you switch directly, the 3D model in the navigation screen will experience a momentary geometric position jump. This visual position jump can easily lead to misjudgment by doctors during delicate operations, and may even trigger conditioned reflex tremors in the hand muscles.

[0160] Therefore, the system uses a time-decay smoothing function to smoothly transition the deformation calculation source from the mechanically predicted incremental deformation displacement field back to the incremental deformation displacement field, thereby eliminating visual position jumps in the navigation screen during cross-modal switching. This smoothing transition process is not completed within a single calculation cycle, but rather within a set time window, achieved through dynamic weighted fusion of the output results from the two data sources. The allocation ratio of the dynamic weights changes exponentially over time, and the specific time-decay smoothing function is defined as follows:

[0161] ;

[0162] In the formula, This represents the overall output displacement vector during the cross-modal switching process, which will be sent to the rendering engine to update the image; This represents the smooth decay rate constant, which determines how quickly the data from the two modes are transferred. This represents the time step since exiting the visual blind spot state, and its value gradually accumulates with the system clock ticks. The mechanically predicted incremental deformation displacement vector represents the last frame before exiting the visual blind zone state, and this vector is latched by the system as the initial reference value for the transition. The incremental deformation displacement vector generated based on feature point matching in the current frame after visual feature recovery represents the result.

[0163] By adjusting this function, in the initial few time steps before exiting the visual blind spot, the decay exponent term is close to 1, and the system still highly trusts the displacement predicted by mechanics; as the time step increases... As the coefficient of motion increases, the decay exponent rapidly approaches zero. At this point, the displacement vector based on real visual observations gradually takes over, until the mechanical influence is completely zero, and the system smoothly returns to control under the visual tracking layer. This smooth transition mechanism in the time domain ensures the continuous evolution of the model topology under any extreme conditions.

[0164] In terms of current technology, traditional neurosurgical navigation systems typically respond to intraoperative endoscopic obstruction by issuing warnings and freezing the current navigation screen, or by crashing the navigation program due to algorithm errors. With these existing systems, surgeons not only lose the ability to observe through the optical endoscope when dealing with emergencies such as sudden massive bleeding, but also the guidance and assistance of digital 3D navigation, turning the surgical area into a safety blind spot.

[0165] The method provided in this embodiment deeply integrates the underlying physical feedback mechanism with the high-level image tracking mechanism across modalities. By dynamically switching to an anatomical semantic-driven biomechanical deformation inference mode, this solution can still reconstruct and maintain a highly realistic dynamic anatomical deformation model through the force and pose sensed by the robot system even in extreme conditions where visual information is lost.

[0166] Furthermore, this solution introduces a mode switching smoothing mechanism based on a time decay function, ensuring that the system maintains smooth and continuous visuals at the boundary between fault occurrence and troubleshooting. Overall, this technical solution significantly improves the robustness of neuroendoscopic robot navigation systems in handling complex and challenging surgical conditions, ensuring the safety and continuity of the surgical procedure, and has extremely important clinical application prospects and practical technical significance.

[0167] Example 3:

[0168] In actual clinical practice in neurosurgery, in addition to slow tissue deformation and sudden bleeding obstruction, there is another physical disturbance that is easily overlooked but seriously affects the safety of surgery: high-frequency mechanical oscillation.

[0169] When surgeons use high-speed power systems (such as skull base drills that rotate at tens of thousands of revolutions per minute) or ultrasonic aspiration scalpels to cut bone and pulverize tumors, the instruments themselves generate intense high-frequency vibrations. This vibrational energy is transmitted to various joints of the system through physical contact and the rigid linkages of the robotic arm. Due to the inherent flexible deformation of the harmonic reducer and the presence of gear backlash, the end of the robotic arm and the neuroendoscope it holds inevitably experience high-frequency minute displacements and slippages.

[0170] This physical-level micro-oscillation directly causes motion blur in the video stream captured by the endoscopic camera. If left unchecked, the navigation screen generated by reprojection will experience high-frequency flickering and severe shaking, leading to visual fatigue for doctors and increasing the risk of misoperation.

[0171] The purpose of this embodiment is to filter out such high-frequency noise from the algorithm's underlying layer without introducing rendering latency by designing a multimodal data fusion filtering state inference algorithm.

[0172] To achieve this technical solution, this embodiment elaborates on the method, which also includes trajectory monitoring and state prediction filtering steps for visual blurring caused by minute slippage and oscillation. Specifically:

[0173] First, in the front-end image acquisition module of the asynchronous slow-track visual layer, the system acquires multiple frames of images from adjacent time nodes continuously output by the neuroendoscopy. The pixel-level local dynamic difference features are extracted through the differential image analysis algorithm, and the visual micro-slip motion vector of the end of the neuroendoscopy in the two-dimensional image plane is calculated.

[0174] Endoscopic cameras typically output high-definition digital image signals at a frame rate of 60 Hz or higher. Within such a dense time sampling interval, two adjacent frames record the physical displacement mapping of the endoscope tip within an extremely short time window. The differential image analysis algorithm first performs grayscale conversion and histogram equalization preprocessing on two frames at adjacent time points to eliminate pixel grayscale interference caused by ambient light flicker.

[0175] Subsequently, the algorithm divides the image into multiple fixed-size pixel macroblocks. Using optical flow or feature point neighborhood-based block matching techniques, it searches for pixel regions in subsequent frames of the time series that have the highest gray-level correlation with the corresponding macroblock in the previous frame. The sub-pixel-level coordinate displacement between the center points of the matched macroblocks in the aforementioned images is defined as the pixel-level local dynamic difference feature. This feature reflects the microscopic flow trend of local image content on the target surface of the imaging sensor.

[0176] When the endoscope lens vibrates, all static tissue textures within the field of view will experience a consistent field-of-view shift with a specific direction and rate. The system performs vector field statistics on all pixel-level local dynamic difference features extracted within the field of view and uses the least squares method to fit a two-dimensional vector that can characterize the relative motion of the global background. This two-dimensional vector is defined as the visual micro-slip motion vector. This vector quantifies the high-frequency deflection of the endoscope's visual axis caused by physical vibration in the visual space dimension.

[0177] Alternatively, since visual image processing inherently involves rendering, transmission, and algorithm calculation time, relying solely on visual data for image stabilization compensation will introduce time lag, causing the compensation actions of the navigation screen to always lag behind the actual physical oscillations, thus producing motion blur.

[0178] In order to obtain the prior oscillation physical information with zero delay, the system synchronously acquires the high-frequency pose data at the fast track communication bus level, performs differential calculation on its historical time series, and extracts the three-dimensional mechanical oscillation vector of the instrument end in physical space.

[0179] The high-frequency pose data is output by the high-frequency fast track control layer at a data refresh rate of kilohertz. It includes the active low-frequency macroscopic motion of the doctor dragging the robotic arm to perform surgical operations, as well as the passive high-frequency microscopic oscillations introduced by external tools.

[0180] To accurately separate unwanted high-frequency oscillations from pose data containing composite frequencies, the system maintains a sliding data window on the time axis to cache a series of recent historical pose sequences. The system performs differential differentiation on this historical time series using a specifically designed digital differential filter. The specific differential stripping and extraction process is implemented through the following mathematical formula:

[0181] ;

[0182] In the formula, This represents the three-dimensional mechanical oscillation vector extracted through stripping; This represents the high-frequency pose data collected in real time under the current time index; The depth dimension parameter representing the sliding data window; This represents the lag time order variable within the sliding data window; The low-pass filter weighting coefficients represent the corresponding time sequence, and the sum of all weighting coefficients is normalized to the value of 1. This represents the current discrete-time index. The summation symbol and its product terms physically constitute a low-pass filter for a weighted moving average, used to estimate and reconstruct a smooth motion trajectory that has had high-frequency components removed.

[0183] Subtracting the smoothed motion trajectory from the current high-frequency pose data yields a difference vector that represents the pure high-frequency perturbation component, which is the extracted three-dimensional mechanical oscillation vector. This vector provides prior information on the acceleration and displacement oscillations of the instrument's end effector in three-dimensional Cartesian physical space with an extremely high update frequency.

[0184] For example, after acquiring visual observation data with time lag but containing true optical mapping characteristics, and physical kinematic data with extremely low latency but containing transmission noise and quantization errors, the system needs to leverage modern control theory to complement the advantages of both. The system constructs a Kalman filter algorithm model for pose trajectory smoothing, inputting the three-dimensional mechanical oscillation vector as a state prediction variable into the state transition equation of the Kalman filter algorithm model, and inputting the visual micro-slip motion vector as an observation update variable into the observation equation of the Kalman filter algorithm model.

[0185] The Kalman filter algorithm is an optimal autoregressive data processing algorithm, particularly suitable for state estimation of linear dynamic systems with multi-source measurement noise. In this model, the state prediction variable reflects the spatial vibration trend of the instrument end effector derived from the underlying motor encoder, which is equivalent to the system's subjective motion expectation; while the observation update variable reflects the displacement of the image content actually captured by the visual sensor, which is equivalent to the system's objective visual feedback.

[0186] The state transition equation utilizes the dynamics of the mechanical system, deriving the prior state estimate and corresponding covariance matrix at the current moment based on the optimal estimate from the previous moment and the current mechanical prediction input. The observation equation establishes a projective geometric mapping from the three-dimensional physical space to the two-dimensional image pixel plane, used to predict the two-dimensional micro-slip vector that should theoretically be observed in the endoscopic image under a given prior state. Through this cross-domain data mapping mechanism, the Kalman filter algorithm model establishes a unified mathematical benchmark for comparison between the state prediction variables and the observation update variables.

[0187] Specifically, the system uses the Kalman filter algorithm model to perform fusion and iterative calculations on the state prediction variables and the observation update variables, filtering out noisy interference data, and calculating the optimal pose state estimate that smoothly approximates the actual physical trajectory of the device's end effector. Within each control cycle, the algorithm subtracts the actual observation update variables obtained from visual calculations from the theoretical observation vectors predicted by the model to obtain the residual, and then performs state correction using the Kalman gain matrix. The update calculation process of the fusion and iterative calculation is derived through the following mathematical formula:

[0188] ;

[0189] In the formula, This represents the optimal pose state estimate calculated after fusion filtering; This represents the prior state quantity derived from the state at the previous moment and the predicted variables of the current state; Represents the Kalman gain matrix; The observed update variable represents the actual calculation obtained based on the difference image analysis; This represents the observation matrix that maps the state of three-dimensional physical space to the two-dimensional image observation plane. The part within square brackets in the formula represents the innovative residual between the actual observation and the theoretical prediction, which represents new information contained in the visual measurement that was not covered by the physical model.

[0190] The value of the Kalman gain matrix determines the weighting of trust between visual observations and physical predictions during the fusion process. When the visual image is severely affected by surgical smoke, leading to inaccurate micro-slip extraction, the covariance of the observation noise increases, and the system automatically reduces the value of the Kalman gain matrix, thereby adopting more of the stable motion trajectory deduced by the physical encoder.

[0191] Conversely, when the transmission system experiences physical data distortion due to flexible deformation, the system increases the trust weight of clear visual features. By continuously performing this dynamic weighted correction, the algorithm can efficiently filter out random white noise and high-frequency interference signals contained in the visual image and mechanical transmission, outputting an estimated value that combines low-latency response characteristics with high stability and anti-interference capability. This value is stripped of microscopic clutter and highly restores the doctor's true active operating intention and the instrument's true macroscopic trajectory, i.e., the optimal pose state estimate.

[0192] like Figure 7 The figure shows the trajectory tracking diagram of multi-source pose Kalman filter under high-frequency micro-oscillation. The horizontal axis represents time in seconds, and the vertical axis represents spatial pose offset in millimeters.

[0193] In the figure, the gray solid line represents the pose trajectory of high-frequency mechanical motion, the green dashed line represents the pose trajectory calculated by delayed vision, and the red solid line represents the optimal pose state estimation trajectory output by Kalman filtering.

[0194] Observing the waveform transformation, it can be found that the gray solid line is accompanied by dense high-frequency oscillating ripples, and the amplitude of its up and down fluctuations reaches about 1.5 mm. This truly reflects the physical and mechanical vibration transmitted to the end of the robotic arm when the surgical instruments are running at high speed.

[0195] Although the green dashed line is relatively smooth, it lags significantly behind the actual macroscopic motion trend on the timeline, exhibiting a phase delay of approximately 0.05 seconds, reflecting the inherent computational time required for visual image acquisition and rendering.

[0196] The red solid line effectively combines the advantages of both, with its waveform smoothly weaving through the oscillation envelope of the gray solid line. This not only filters out high-frequency noise from mechanical feedback but also overcomes the time lag of visual data, closely tracking the real, smooth motion trajectory.

[0197] However, careful observation of the red solid line in the peak and trough regions around 0.5 and 1.5 seconds reveals a slight amplitude overshoot and minimal high-frequency residual fluctuations. This local phenomenon objectively indicates that under extreme dynamic conditions such as sudden stops or reversals with high acceleration, the adaptive adjustment mechanism of the Kalman gain matrix in the current filtering algorithm generates a certain amount of numerical integral inertia, leading to a small transient deviation between the state prediction and the actual physical position. This suggests that the system still has room for further optimization in handling the dynamic distribution of observation noise under highly dynamic nonlinear dynamic environments.

[0198] It is also important to note that after successfully extracting the clean, smooth, and latency-free optimal trajectory baseline, the system needs to apply it to the final computer graphics rendering pipeline to achieve visual stabilization of the screen image.

[0199] The system updates the optimal pose state estimate as the pose compensation benchmark into the computation flow of the projection matrix, and uses the optimal pose state estimate to perform inverse compensation and cancellation on the offset coordinates during the reprojection process of the three-dimensional mesh vertices, so as to suppress the visual oscillation of the fused navigation screen.

[0200] In the technical solution of Embodiment 1, when the vertices of the 3D panoramic anatomical mesh model are projected onto a 2D screen, they rely on a projection matrix containing high-frequency pose data and visually calculated pose. If the unfiltered raw pose data is used directly for projection calculation, even if the relative deformation calculation of the anatomical model itself is very accurate, the entire 3D mesh as a whole will undergo high-frequency vertical and horizontal translational jumps in the screen coordinate system.

[0201] In the compensation mechanism provided in this embodiment, the computation flow of the projection matrix introduces reverse cancellation logic. The system performs matrix inversion and subtraction operations on the original oscillation pose and the optimal pose state estimate obtained after filtering to generate a reverse bias transformation matrix with a direction completely opposite to the physical high-frequency disturbance and an amplitude absolutely equal to it.

[0202] When the vertices of the 3D mesh are projected onto the 2D pixel plane of the screen through a projection matrix that incorporates the inverse bias transformation matrix, the image coordinate offset component caused by the high-frequency oscillation of the physical device is precisely neutralized by the inverse bias. For the surgeon observing the surgical process in front of the screen, the effect is that although the drill in their hand is generating strong physical vibrations and the endoscope lens is also experiencing slight physical oscillations within the narrow cranial cavity, the 3D panoramic anatomical model and the fused navigation screen appear as if they are fixed on a digitally stabilized gimbal. The center focus of the image remains stably maintained on the surface of a specific anatomical structure, and the mesh deformation at the edges is smooth and seamless, without any visual high-frequency jitter or ghosting lag.

[0203] In terms of existing surgical robot navigation technology, most systems address image blurring and positioning jitter caused by high-frequency oscillations by adding simple low-pass or mean filters to the software front end. While these traditional filtering algorithms can smooth the jitter curve, they inherently sacrifice the system's phase response characteristics, inevitably introducing display delays of tens or even hundreds of milliseconds. In the high-precision grinding operations of neurosurgery, this lag in the navigation image can cause temporal and spatial deviations between the surgeon's hand movements and visual feedback, easily leading to over-grinding and damage to deep nerves and blood vessels.

[0204] The multimodal state prediction filtering scheme provided in this embodiment overcomes the limitations of filtering from a single data source. It utilizes kilohertz-level high-frequency mechanical kinematic data as the prior dynamic model for state prediction, compensating for the inherent time delay in visual processing. Simultaneously, it uses high-precision two-dimensional visual micro-slip features as observation correction data, eliminating the accumulation of flexibility errors in the mechanical transmission system. This iterative mechanism of alternating state prediction and observation updates within the Kalman filtering framework enables the navigation system to extract and cancel high-frequency vibration noise parasitic on effective movements without delay within an extremely short control cycle.

[0205] In summary, this technical solution, together with the dynamic partitioning deformation correction in Embodiment 1 and the force and visual blind spot compensation in Embodiment 2, effectively solves the problem of interference to spatial positioning caused by dynamic and complex interference at the underlying signal link level, and greatly improves the operational safety and system robustness of the neuroendoscopic robot in delicate and high-intensity surgical scenarios.

[0206] Example 4:

[0207] like Figure 8 As shown, this embodiment provides a system embodiment corresponding to the aforementioned method embodiment.

[0208] In the context of minimally invasive neuroendoscopic robotic surgery, existing navigation systems typically rely on static preoperative medical images (such as CT or MRI) for rigid spatial registration. However, during actual surgery, as lesions are removed, cerebrospinal fluid is lost, and surgical instruments exert physical traction, brain tissue undergoes significant non-rigid deformation, leading to spatial deviations between the static image model and the actual intraoperative anatomical structure.

[0209] If real-time endoscopic video streams are directly introduced for dense 3D reconstruction of the entire field of view to solve this problem, the massive matrix computation can easily overwhelm system resources, causing thread blocking with the underlying high-frequency motion control process of the robotic arm. This can lead to serious technical problems such as navigation screen delays or loss of robotic arm control signals. Furthermore, performing indiscriminate deformation calculations on the panoramic view cannot effectively distinguish the stress areas in contact with the instrument, resulting in insufficient accuracy in representing the deformation of locally compressed areas.

[0210] To address the aforementioned technical issues, this embodiment provides a neuroendoscopic robot navigation system integrating AI-powered visual panoramic stitching, applicable to a surgical robot system comprising a neuroendoscope, a six-degree-of-freedom robotic arm, and a computing unit, specifically including:

[0211] The parameter acquisition module is used to obtain the initial registration transformation matrix between the patient's world coordinate system and the robotic arm's base coordinate system, as well as the camera's intrinsic parameter matrix. In practical hardware applications, this module primarily interacts with the binocular infrared optical tracking system (such as the NDI optical navigator) and calibration tools within the operating room. By reading the reflective marker array fixed on the patient's head fixation frame and the markers on the robotic arm base, the system establishes a unified three-dimensional world coordinate system. Simultaneously, using a specific calibration plate to perform multi-angle imaging in front of the endoscope lens, this module calculates the focal length, optical center position, and distortion parameters of the optical lens, thereby establishing a mapping relationship between the abstract mathematical matrix and the actual physical imaging equipment and spatial position.

[0212] The dual-track architecture module is used to construct a dual-track decoupled navigation architecture, comprising an asynchronous slow-track vision layer and a high-frequency fast-track control layer. In the actual computing unit hardware configuration, this module operates through the underlying operating system kernel scheduling mechanism. The system's computing unit includes an industrial control computer running a real-time operating system (RTOS) and a graphics workstation configured with a high-performance parallel computing architecture (GPU). The dual-track architecture module deploys the logic of the high-frequency fast-track control layer in the isolated CPU core of the industrial control computer, ensuring the high priority and clock synchronization characteristics of the control signals; simultaneously, it directs the data flow of the asynchronous slow-track vision layer to the GPU's video memory for tensor parallel computation. This module avoids the obstruction of control feedback by visual computation at the system's underlying architecture level.

[0213] The slow-track vision processing module is used to enable the asynchronous slow-track vision layer to process the neuroendoscopic video stream, generate a three-dimensional panoramic anatomical mesh model of the intracranial cavity through depth estimation and mesh reconstruction, and output anatomical semantic labels, visually calculated pose and global rigid deformation correction matrix.

[0214] In clinical settings, the neuroendoscopy unit outputs high-definition surgical video signals at a frame rate of 60Hz or higher. This module receives the video stream via an image acquisition card and runs a pre-trained convolutional neural network model using the GPU's built-in deep learning acceleration engine. After extracting image features frame by frame, the network model outputs semantic labels with anatomical tissue classifications (such as blood vessels, nerves, and brain tissue entities) and calculates the relative visual motion pose of the endoscope lens based on optical flow tracing. Subsequently, the module calculates the overall sedimentation of brain tissue due to cerebrospinal fluid loss—the global rigidity deformation correction matrix—using a point cloud registration algorithm.

[0215] The high-frequency fast track control and calculation module is used to enable the high-frequency fast track control layer to collect the state data of the robotic arm, calculate the real-time pose coordinates of the end effector based on the kinematic model, and output high-frequency pose data with real-time confidence labels. In the actual electromechanical system, each joint of the six-degree-of-freedom robotic arm integrates a high-precision absolute encoder and a servo driver.

[0216] This module reads joint angle data at a sampling frequency of over 1000Hz via an industrial Ethernet bus (such as the EtherCAT protocol). Using a built-in DH forward kinematics algorithm processor, the module calculates the precise coordinates of the surgical instrument's end effector in three-dimensional space in real time. Simultaneously, the module's internal safety monitoring unit assesses hardware physical feedback states such as servo motor current fluctuations and transmission backlash, attaching quantified real-time confidence labels to the output pose data to characterize the reliability of the current physical data.

[0217] The dynamic partitioning module is used to dynamically divide the 3D panoramic anatomical mesh model into a core operation area, a transition area, and a global reference area based on the real-time pose coordinates. At the memory data management level, this module serves as the central hub for computing power allocation. Since non-rigid deformation is concentrated in the physical contact area at the instrument's end effector, this module compares the instrument spatial coordinates output by the fast track control solution module with the coordinate set of the anatomical mesh vertices in real time.

[0218] By setting a spatial distance threshold, the grid region closest to the instrument tip and containing high-risk anatomical semantic tags is indexed as the core operating area. This module directly addresses the inefficiency caused by panoramic indiscriminate computation in the background technology, achieving spatial dimension reduction in computation.

[0219] The local correction calculation module is used to initiate local non-rigid incremental correction for the core operating area, extract image feature points and perform inter-frame matching to construct and solve the non-rigid deformation energy function, generating an incremental deformation displacement field. This module utilizes the parallel floating-point computing power of the GPU, concentrating computing power on the core operating area selected by the dynamic partitioning module. The module establishes a mathematical energy function that includes data fitting error and Laplacian smoothing constraints by tracking the microscopic texture displacement of the tissue surface within the core operating area. Solving this equation using the Gauss-Newton iterative method, the module outputs a three-dimensional deformation vector set (i.e., the incremental deformation displacement field) that accurately expresses the local tissue indentation and rebound caused by instrument compression or traction.

[0220] The image fusion projection module combines the global rigid deformation correction matrix and the incremental deformation displacement field to calculate the comprehensive deformation coordinates of the mesh vertices using smoothing weighting coefficients. Based on the high-frequency pose data and the visually calculated pose, it constructs a projection matrix to reproject the 3D mesh vertices onto a 2D image plane, generating a fused navigation screen. This module ultimately interfaces with a medical 4K display terminal in the operating room.

[0221] In the graphics rendering pipeline, this module utilizes smoothing weights in the transition zone to smoothly transition and stitch together drastic local deformations and global rigid displacements, preventing topological tearing in the 3D anatomical model. Finally, the module generates a viewpoint matrix by fusing visual and physical pose data based on confidence labels, and renders the deformed 3D coordinate mapping into a 2D augmented reality navigation video stream that doctors can directly view.

[0222] In practical applications, this embodiment constructs a dual-track decoupled architecture comprising an asynchronous slow-track vision layer and a high-frequency fast-track control layer. This allows the time-consuming 3D image reconstruction process and the low-latency robotic arm pose calculation process to run independently within the physical computing unit. This structure maintains continuous high-frequency monitoring of the robotic arm's state while fundamentally preventing sudden large-scale visual calculations from blocking the robot's underlying control feedback commands.

[0223] Furthermore, relying on the spatial filtering mechanism of the dynamic partitioning module and the local correction calculation module, the system dynamically divides the three-dimensional anatomical mesh into regions based on the real-time spatial coordinates of the instrument's end effector. This concentrates limited computing resources on the key anatomical regions (core operating areas) where the instrument directly interacts, allowing for the separate solution of non-rigid deformation energy functions. This centralized computing power scheduling strategy not only improves the computational efficiency of local deformation tracking but also enhances the accuracy of deformation representation in the instrument's operating area.

[0224] Ultimately, by integrating deformation data and reprojecting it through the image fusion projection module, the navigation system can output images that reflect the deformation state of soft tissue under the actual physical action of instruments in a coherent and stable manner. This provides operators with an accurate reference for judging the spatial physical relationship between instrument positions and high-risk anatomical structures, effectively improving the safety of surgical procedures and the reliability of equipment operation.

[0225] Example 5:

[0226] Corresponding to the above embodiments, the present invention also proposes an electronic device.

[0227] like Figure 9 The diagram shows a structural schematic of an electronic device according to the present invention. The electronic device 100 includes a processor 101 and a memory 103. The processor 101 and the memory 103 are connected, for example, via a bus 102. Optionally, the electronic device 100 may further include a transceiver 104. It should be noted that in practical applications, the transceiver 104 is not limited to one unit, and the structure of this electronic device 100 does not constitute a limitation on the embodiments of the present invention.

[0228] Processor 101 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 101 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0229] Bus 102 may include a pathway for transmitting information between the aforementioned components. Bus 102 may be a PCI bus or an EISA bus, etc. Bus 102 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0230] The memory 103 stores a computer program corresponding to the neuroendoscopic robot navigation method with fusion AI visualization panoramic stitching as described in the above embodiments of the present invention. This computer program is controlled and executed by the processor 101. The processor 101 executes the computer program stored in the memory 103 to implement the content shown in the aforementioned method embodiments.

[0231] Among them, electronic devices 100 include, but are not limited to: mobile terminals such as laptops and PADs (tablet computers) and fixed terminals such as desktop computers. Figure 9 The electronic device 100 shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0232] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A neuroendoscopic robot navigation method integrating AI-based visual panoramic stitching, applied to a surgical robot system comprising a neuroendoscope, a six-degree-of-freedom robotic arm, and a computing unit, characterized in that... include: Obtain the initial registration transformation matrix between the patient's world coordinate system and the robot arm's base coordinate system, as well as the camera intrinsic parameter matrix; Construct a dual-track decoupled navigation architecture, including an asynchronous slow track visual layer and a high-frequency fast track control layer; The asynchronous slow-track visual layer processes the neuroendoscopic video stream, generates a three-dimensional panoramic anatomical mesh model of the intracranial cavity through depth estimation and mesh reconstruction, and outputs anatomical semantic labels, visually calculated pose, and global rigid deformation correction matrix. The high-frequency fast track control layer collects the state data of the robotic arm, calculates the real-time pose coordinates of the end effector based on the kinematic model, and outputs high-frequency pose data with real-time confidence labels. Based on the real-time pose coordinates, the three-dimensional panoramic anatomical mesh model is dynamically divided into a core operation area, a transition area, and a global reference area. Local non-rigid incremental correction is initiated for the core operation area, image feature points are extracted and inter-frame matching is performed to construct and solve the non-rigid deformation energy function and generate an incremental deformation displacement field. By combining the global rigid deformation correction matrix and the incremental deformation displacement field, the comprehensive deformation coordinates of the mesh vertices are calculated through smoothing weight coefficients. Based on the high-frequency pose data and the visually calculated pose, a projection matrix is ​​constructed to reproject the three-dimensional mesh vertices onto the two-dimensional image plane to generate a fused navigation screen.

2. The method according to claim 1, characterized in that, The asynchronous slow-track visual layer processes the neuroendoscopic video stream, including: Multi-scale partitioning preprocessing is performed on a single frame of endoscopic image, and a pixel-level dense depth map and the anatomical semantic labels are output through a depth estimation network; A preoperative reference point cloud is obtained, and an intraoperative reconstructed point cloud is generated based on the extracted feature point 3D coordinates. A rigid registration algorithm is used to minimize the sum of squared Euclidean distances between the preoperative reference point cloud and the intraoperative reconstructed point cloud, and the global rigid deformation correction matrix is ​​obtained by solving the algorithm.

3. The method according to claim 1, characterized in that, The high-frequency fast track control layer outputs high-frequency pose data with real-time confidence labels, including: Real-time acquisition of four types of interference data: joint transmission clearance fluctuation, motor temperature drift, electromagnetic interference intensity, and physiological displacement amplitude; The measured values ​​of various types of interference data are divided by their corresponding critical thresholds, and then weighted and summed using weighting factors to obtain the real-time interference deviation value. Based on the interval distribution characteristics of the real-time interference deviation values, they are discretely mapped to real-time confidence labels of different numerical levels.

4. The method according to claim 1, characterized in that, The dynamic division of the three-dimensional panoramic anatomical mesh model into a core operation area, a transition area, and a global reference area includes: Calculate the Euclidean distance from any vertex of the three-dimensional panoramic anatomical mesh model to the end of the instrument; If the Euclidean distance is less than or equal to the first distance threshold, the corresponding mesh vertex is assigned to the core operation area; If the Euclidean distance is greater than the first distance threshold and less than or equal to the second distance threshold, the corresponding mesh vertex is assigned to the transition area; If the Euclidean distance is greater than the second distance threshold, the corresponding mesh vertex is assigned to the global reference region; Wherein, the second distance threshold is greater than the first distance threshold; Simultaneously, based on the aforementioned anatomical semantic tags, anatomical structures characterized as high-risk are forcibly included in the core operating area.

5. The method according to claim 1, characterized in that, The step of extracting image feature points and performing inter-frame matching to construct and solve the non-rigid deformation energy function and generate an incremental deformation displacement field includes: Sparse feature points are extracted within the core operation area and inter-frame matching is performed. After removing false matches, a set of valid three-dimensional feature point pairs is obtained. Construct a total energy function that includes a data fitting error term and a smoothing constraint term, wherein the data fitting error term is used to constrain the coordinate deviation of the matching feature points before and after deformation, and the smoothing constraint term is constructed based on the normalized Laplacian matrix of the three-dimensional panoramic anatomical mesh model. The incremental deformation displacement field of the grid vertices within the core operating region is obtained by solving for the deformation parameters that minimize the total energy function.

6. The method according to claim 1, characterized in that, The step of calculating the comprehensive deformation coordinates of mesh vertices through smoothing weighting coefficients and constructing a projection matrix based on the high-frequency pose data and the visually calculated pose includes: For the mesh vertices of the core operation area and the global reference area, fixed smoothing weight coefficients are assigned respectively; for the mesh vertices of the transition area, a cosine smoothing transition function is used to calculate their dynamic smoothing weight coefficients. By combining the camera intrinsic parameter matrix and the initial registration transformation matrix, and using the real-time confidence label as an adjustment weight, the high-frequency pose data and the visually calculated pose are weighted and fused to generate the projection matrix.

7. The method according to claim 1, characterized in that, The method also includes a triple cross-validation step based on both global and local dimensions, specifically including: Using the high-frequency pose data, the visually calculated pose, and the real-time confidence labels as independent data sources, the global pose deviation is obtained by calculating the matrix norm. Local deformation deviation is obtained by calculating the Euclidean distance between the coordinates of the instrument end obtained from visual reconstruction and the coordinates of the physical instrument end. Based on the magnitude of the global pose deviation and the local deformation deviation, a graded safety response mechanism of the corresponding level is triggered.

8. The method according to claim 7, characterized in that, The triggering of the corresponding level of graded security response mechanism includes: When both the global pose deviation and the local deformation deviation are less than or equal to the warning threshold, the system operates normally with all functions. When the global pose deviation or the local deformation deviation is greater than the warning threshold and less than the braking threshold, the system is determined to enter the warning state, the navigation interface is controlled to output the corresponding level of alarm prompt, and the maximum movement speed of the robotic arm is adaptively limited, while the local deformation correction frequency of the core operation area is forcibly increased. When the global pose deviation or the local deformation deviation is greater than or equal to the braking threshold, the system determines that it enters the braking state, triggers the flexible braking of the robotic arm, and locks the current position of the endoscope. Wherein, the braking threshold is greater than the warning threshold.

9. The method according to claim 1, characterized in that, In the step of initiating local non-rigid incremental correction for the core operating area, the local deformation update frequency is positively correlated with the real-time confidence label. When the real-time confidence label is the first confidence level, local deformation update is performed at the first preset frequency; When the real-time confidence label is the second confidence or the third confidence, the local deformation update is performed accordingly at the second preset frequency or the third preset frequency. Wherein, the first confidence level is greater than the second confidence level, and the second confidence level is greater than the third confidence level; the first preset frequency is greater than the second preset frequency, and the second preset frequency is greater than the third preset frequency.

10. The method according to claim 1, characterized in that, The process of reprojecting onto a two-dimensional image plane to generate a fused navigation screen specifically employs a hierarchical multi-layer fusion rendering mechanism, including: From bottom to top, a 3D panoramic anatomical base layer, a real-time high-definition semi-transparent endoscope overlay layer, a navigation accuracy heat map layer, and a graded early warning prompt layer are superimposed sequentially. The navigation accuracy heatmap maps three regions corresponding to the dynamic partitions, with accuracy levels mapped and labeled. The core operation area is labeled with the first accuracy level, the transition area with the second accuracy level, and the global reference area with the third accuracy level.

Images