Real-time surgical navigation system and method based on neuromorphic calculation and physical constraint

Abstract

The invention discloses a real-time surgical navigation system and method based on neuromorphic calculation and physical constraint, and relates to the technical field of computer-aided surgeries, and the system comprises a preoperative planning module which obtains patient volume data, generates a digital twin model and a surgical planning report, and sets biomechanical parameters; the neuromorphic sensing module is used for acquiring an original video frame and extracting to obtain anatomical landmark point displacement; the liquid state self-adaptive module is used for obtaining multiple domain deviation intensities based on the visual feature vectors; the pulse auxiliary optimization module is used for obtaining a measurement pulse sequence and a prediction state based on multi-source sensor data and obtaining a camera-organ relative pose based on the data; the physical constraint compensation module is used for obtaining a grid model based on the camera-organ relative pose, the anatomical landmark point displacement, the digital twinborn model and the biomechanical parameters; and the augmented reality visual module is used for obtaining a colored grid mask based on the grid model, superposing the colored grid mask with the original video frame and outputting a superposed video frame. And the operation safety and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of computer-assisted surgery technology, and more specifically to a real-time surgical navigation system and method based on neuromorphic computing and physical constraints. Background Technology

[0002] Currently, image-guided surgery (IGS) is an important technology in modern precision medicine. By registering preoperative medical images with real-time intraoperative images, it provides surgeons with visual guidance on anatomical structures and lesion locations. However, existing surgical navigation systems have the following technical limitations: 1) High computational latency: For example, traditional visual perception modules based on convolutional neural networks (CNNs) struggle to achieve end-to-end latency below 50 milliseconds on edge devices. This limits their application in real-time surgical procedures, potentially leading to delayed visual feedback and increasing surgical risks. 2) Insufficient soft tissue deformation compensation: Soft tissues such as the liver undergo significant deformation during surgery due to respiratory movements, instrument pressure, and gravity. Existing systems often employ rigid registration or simple free deformation (FFD) methods, lacking biomechanical constraints. This results in registration errors accumulating as the surgery progresses, with target registration error (TRE) often exceeding 5 millimeters, impacting surgical safety. 3) Weak domain adaptability: Surgical scenarios frequently involve smoke, bleeding, sudden changes in lighting, and tissue deformation. Interference factors such as occlusion cause a distribution shift between the training domain and the actual surgical domain in traditional deep learning models, resulting in a sharp decline in model performance. This necessitates system downtime for retraining or manual parameter adjustment, making it impossible to achieve online adaptation during surgery. 4) Low accuracy of multi-sensor fusion: Existing systems often use heuristic weight allocation for the fusion of multi-source sensor data such as endoscopic vision, inertial measurement unit (IMU), and mechanical encoder, failing to dynamically adjust the fusion strategy according to the real-time scenario. This results in insufficient robustness when some sensors fail or noise increases. 5) High energy consumption and hardware cost: Traditional CNN models have a large number of parameters and high computational complexity, requiring high-power GPU support. This is not conducive to the lightweight deployment of mobile surgical vehicles or wearable devices, limiting the clinical promotion of surgical navigation systems.

[0003] Therefore, how to compensate for soft tissue deformation in real time in complex surgical environments, thereby improving surgical safety and precision, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a real-time surgical navigation system and method based on neuromorphic computing and physical constraints that overcomes or at least partially solves the above problems, thereby compensating for soft tissue deformation in real time in complex surgical environments and improving surgical safety and accuracy.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a real-time surgical navigation system based on neuromorphic computing and physical constraints, comprising: The preoperative planning module is used to acquire the target patient's body data, generate multi-label segmentation masks, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters; The neuromorphic perception module is used to acquire raw endoscopic video frames and extract visual feature vectors and raw pulse sequences, and obtain the displacement of anatomical landmarks based on the raw pulse sequences. A liquid adaptive module is used to obtain the deviation intensity of multiple domains based on the visual feature vector; The pulse-assisted optimization module is used to acquire multi-source sensor data during surgery and obtain a measurement pulse sequence. Based on the domain deviation intensity and the state vector at the previous moment, the predicted state at the current moment is obtained, and the camera-organ relative pose is obtained based on the predicted state, the original pulse sequence, and the measurement pulse sequence. The physical constraint compensation module is used to obtain the transformed displacement based on the relative pose of the camera and organ and the displacement of the anatomical landmarks, construct the total loss function based on the three-dimensional digital twin model, the transformed displacement and the biomechanical parameters, and optimize the variable field neural network to obtain a three-dimensional mesh model. The augmented reality visualization module is used to obtain a two-dimensional mask based on the camera-organ relative pose and the three-dimensional mesh model, assign corresponding colors to different anatomical structures and combine them with the two-dimensional mask to obtain a colored mesh mask, and superimpose it with the original endoscopic video frame to output an AR superimposed video frame, and provide real-time distance warning based on the surgical planning report.

[0006] In one embodiment, the preoperative planning module includes: A medical image loading unit is used to acquire CT or MRI data of the target patient as the volume data; An automatic segmentation unit is used to segment related organ structures and lesion structures based on the volume data, and generate the corresponding multi-label segmentation mask; The three-dimensional reconstruction unit is used to extract the three-dimensional mesh surface of each anatomical structure based on the multi-label segmentation mask, generate a triangular mesh, and perform smoothing and simplification processing based on the triangular mesh to obtain the three-dimensional digital twin model of the target patient. The surgical planning unit is used to set the resection safety boundary and perform three-dimensional expansion with the multi-label segmentation mask, calculate the resection volume, estimate blood loss, analyze the minimum distance between the tumor and blood vessels / bile ducts, and generate safety distance information based on the minimum distance to form the surgical planning report. A physical prior unit is used to set the biomechanical parameters according to the organ and tissue type.

[0007] In one embodiment, the neuromorphic perception module includes: A visual feature acquisition unit is used to acquire the original endoscopic video frame and input it into the pulse backbone neural network of the first pulse time encoding to obtain the visual feature vector. The original pulse acquisition unit is used to obtain the original pulse sequence based on the firing time recorded by the leaky integral firing neurons in the pulse backbone neural network; The anatomical displacement determination unit is used to obtain a pulse feature map based on the original pulse sequence, track key anatomical structures across multiple frames based on the pulse feature map, and obtain the displacement of the anatomical landmarks according to the location of the landmarks in the previous frame and the location of the maximum response point in the current frame.

[0008] In one embodiment, the liquid adaptive module includes: The offset intensity determination unit is used to detect various domain offset intensities by inputting the data feature vector into a time-varying weighted liquid neural network and using a domain offset detection controller. The dynamic parameter tuning unit is used to dynamically adjust the confidence threshold, measurement noise covariance, and convergence tolerance in the pulse-assisted optimization module and the physical constraint compensation module based on the domain deviation intensity, so as to obtain the adjustment parameters.

[0009] In one embodiment, the pulse-assisted optimization module includes: The parameter conversion unit is used to convert the domain deviation intensity into fusion adjustment parameters; The data acquisition unit is used to acquire visual measurement data, IMU measurement data, and encoder measurement data during surgery as the multi-source sensor data. The prediction output unit is used to input the state vector of the previous moment and the fusion adjustment parameters into the constant velocity motion model to obtain the predicted state at the current moment. A multi-source data encoding unit is used to encode the multi-source sensor data to obtain the measurement pulse sequence; The gain matrix acquisition unit is used to input the pulse gain network after splicing the measured pulse sequence, the original pulse sequence, and the pulse sequence of the predicted state to obtain a continuous value gain matrix; The state update unit is used to update the predicted state and the measurement values ​​corresponding to the multi-source sensor data based on the continuous value gain matrix, remove outliers with Mahalanobis distance exceeding the threshold, and obtain the camera-organ relative pose.

[0010] In one embodiment, the physical constraint compensation module includes: A coordinate system transformation unit is used to transform the displacement of the anatomical landmark from the camera coordinate system back to the organ coordinate system based on the camera-organ relative pose, so as to obtain the transformed displacement. The displacement vector acquisition unit is used to input the vertex coordinates of the three-dimensional digital twin model into the deformation field neural network to obtain the three-dimensional displacement vector; The function parameter acquisition unit is used to construct a hyperelastic material model based on the biomechanical parameters, obtain the strain energy density function, and obtain the Cauchy stress tensor based on the strain energy density function. The loss function construction unit is used to construct the total loss function, which includes multiple boundary condition constraints, based on the transformed displacement and the Cauchy stress tensor. The model optimization unit is used to optimize the deformable field neural network based on the total loss function through an optimizer to obtain the deformed three-dimensional digital twin model and the three-dimensional mesh model.

[0011] In one embodiment, the augmented reality visual module includes: A camera projection unit is used to construct a projection matrix based on the camera-organ relative pose, and to project the vertices of the three-dimensional mesh model onto a two-dimensional image plane based on the projection matrix and the camera intrinsic parameter matrix. The grid rasterization unit is used to perform rasterization processing of triangular grids based on the two-dimensional image plane using a depth buffer algorithm to generate a two-dimensional mask of the organ surface. Anatomical structure color matching unit, used to assign a unique corresponding color to different anatomical structures to obtain a color matching scheme; A coloring unit is used to color the two-dimensional mask based on the color scheme to obtain the color mesh mask; An overlay display unit is used to overlay the original endoscope video frame with the shading mesh mask and output the AR overlay video frame. The safety distance warning unit is used to extract the tumor resection safety boundary distance, the minimum safety distance of major blood vessels, and the minimum safety distance of bile duct as safety thresholds based on the surgical planning report, and to obtain the Euclidean distance between the tip of the surgical instrument and the tumor boundary, the surface of the blood vessel, and the bile duct in real time. When the distance is less than the corresponding safety threshold, a warning is issued.

[0012] In one embodiment, the augmented reality visualization module further includes: A head-up display unit is used to overlay and display key performance indicators on the AR overlay video frame; The key performance indicators include: real-time frame rate, latency of each module, total latency, domain offset strength, state estimation confidence, and current minimum safe distance.

[0013] In one embodiment, it further includes: a failure safety monitoring module; The failure safety monitoring module is used to monitor and determine in real time whether the outputs of the neuromorphic perception module, the liquid adaptive module, the pulse-assisted optimization module, and the physical constraint compensation module are abnormal. When the neuromorphic perception module outputs an abnormality: the impulse sparsity is >0.5 or the variance of the output feature vector is abnormal, the module switches to the MobileNet or EfficientNet model for feature extraction. When the liquid adaptive module outputs an abnormal value (domain offset > 0.9) or the confidence modulation module outputs an abnormal negative value, the adaptive parameters are frozen and the system reverts to the initialized fixed parameters. When the pulse-assisted optimization module outputs an abnormality: the Kalman gain does not meet the positive definiteness or generates a numerical overflow, it automatically switches to the classic extended Kalman filter (EKF) or unscented Kalman filter (UKF) and uses a fixed gain calculation formula. When the physical constraint compensation module outputs an abnormality: the physical residual exceeds the convergence tolerance or the optimization iteration fails to converge, it will revert to rigid registration or use the pre-calculated average deformation template. Each module output includes a confidence score, and a warning is displayed when the global confidence score is less than 0.5.

[0014] Secondly, embodiments of the present invention provide a real-time surgical navigation method based on neuromorphic computing and physical constraints, comprising: Obtain the target patient's body data to generate a multi-label segmentation mask, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters; The original endoscopic video frames are acquired and visual feature vectors and original pulse sequences are extracted. The displacement of anatomical landmarks is obtained based on the original pulse sequences. Multiple domain deviation intensities are obtained based on the aforementioned visual feature vectors; Acquire multi-source sensor data during surgery and obtain a measurement pulse sequence. Based on the domain deviation intensity and the state vector at the previous moment, the predicted state at the current moment is obtained. Based on the predicted state, the original pulse sequence, and the measurement pulse sequence, the camera-organ relative pose is obtained. The transformed displacement is obtained based on the relative pose of the camera and organ and the displacement of the anatomical landmarks. The total loss function is constructed based on the three-dimensional digital twin model, the transformed displacement and the biomechanical parameters, and the variable field neural network is optimized to obtain a three-dimensional mesh model. A two-dimensional mask is obtained based on the camera-organ relative pose and the three-dimensional mesh model. A corresponding color is assigned to the two-dimensional mask based on different anatomical structures to obtain a colored mesh mask. This mask is then superimposed on the original endoscopic video frame to output an AR superimposed video frame. Real-time distance warnings are provided based on the surgical planning report.

[0015] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a real-time surgical navigation system and method based on neuromorphic computing and physical constraints, which has the following beneficial effects: 1. Low Latency: This invention, based on a first-pulse time-encoded spiking neural network, achieves a significant reduction in neuron sparse activation rate, with computational and storage access requirements only 1 / 10 that of traditional CNNs. Through optimized perceptual layer design, latency is significantly reduced to below 10 milliseconds, a reduction of 30-50 milliseconds compared to traditional techniques. Furthermore, employing a five-layer pipelined parallel execution strategy, the end-to-end total latency is controlled within 50 milliseconds, meeting the stringent timing requirements of real-time surgical navigation and effectively solving the problem of visual feedback lag caused by excessive computational latency in existing technologies.

[0016] 2. High-precision deformation compensation: Using an H¹ stabilized physical information neural network, the biomechanical equation (NeoHookean or linear elastic model) is embedded as a strongly constrained loss function to ensure that the deformation field satisfies the force balance equation and material constitutive relation. The target registration error is greatly reduced, and the accuracy is greatly improved compared with the traditional FFD method, which significantly improves the safety and precision of the operation.

[0017] 3. Employing Liquid Neural Networks: As a novel neural network algorithm in AI / ML research, it possesses the compactness and dynamism of time-series prediction. Its time-varying weight mechanism and continuous-time dynamics allow the network to rapidly adapt to domain shifts such as smoke, bleeding, and sudden changes in illumination in surgical scenes within 20 frames (approximately 0.67 seconds), without requiring downtime for retraining or manual parameter adjustments. This adaptive capability enables a domain shift detection accuracy exceeding 90%, effectively solving the problem of drastic performance degradation of deep learning models under distributed shifts in existing technologies.

[0018] 4. Robust Multi-Sensor Fusion: This invention employs a spiking neural network to learn an adaptive Kalman gain matrix, replacing the traditional fixed gain formula. It dynamically adjusts the fusion weights based on the real-time sensor noise statistics, and combines this with Mahalanobis distance outlier detection to achieve a 30% outlier tolerance. Even with partial sensor failure or increased noise, it maintains high-precision state estimation with position errors less than 2 mm and attitude errors less than 1 degree. For example, by constructing a multi-sensor fusion model and using a radial basis function (RBF) network to fuse the outputs of a magnetic sensor and a temperature sensor, the accuracy and stability of the detection system can be significantly improved.

[0019] 5. Low power consumption: The event-driven and sparse activation characteristics of spiking neural networks can reduce power consumption to less than 1 watt when deployed on neuromorphic hardware, which is 20-50 times lower than the 20-50 watt power consumption of traditional CNNs on GPUs. This supports long-term battery life for mobile surgical vehicles and wearable devices, reducing hardware costs and the threshold for clinical promotion.

[0020] 6. Physical interpretability: The PINN model enforces biomechanical equations through a physical loss function, and the predicted deformation field has clinical interpretability. Doctors can understand and trust the system output, which meets the requirements of medical device regulation for algorithm transparency and verifiability. Compared with purely data-driven black box models, it is easier to obtain clinical recognition and regulatory approval.

[0021] 7. Multi-level failure safety protection: The five-layer processing network is designed with independent failure detection and rollback mechanisms, as well as a global confidence propagation and alarm system. When any module is abnormal, it automatically switches to the classic algorithm or degradation mode to avoid system crashes caused by single point of failure and maximize the safety of the operation.

[0022] 8. Flexible deployment: Through Docker containerization and ROS2 interface, it supports flexible deployment and interoperability between edge devices, cloud servers and robotic surgical platforms, adapting to the hardware conditions and workflows of different medical institutions. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the structure of a real-time surgical navigation system based on neuromorphic computing and physical constraints provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the pulsed convolutional backbone network structure provided in an embodiment of the present invention.

[0026] Figure 3 This is a flowchart of the multi-sensor fusion process provided in an embodiment of the present invention.

[0027] Figure 4 This is a schematic diagram of the decision tree and fallback strategy of the failure safety mechanism provided in the embodiments of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Example 1 like Figure 1 As shown in the figure, this invention discloses a real-time surgical navigation system based on neuromorphic computing and physical constraints, including: a preoperative planning module, a neuromorphic perception module, a liquid adaptive module, a pulse-assisted optimization module, a physical constraint compensation module, and an augmented reality visualization module.

[0030] The preoperative planning module is used to acquire the target patient's body data, generate multi-label segmentation masks, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters.

[0031] Furthermore, the preoperative planning module includes: The medical image loading unit is used to acquire CT or MRI data of the target patient as volume data. An automatic segmentation unit is used to segment related organ structures and lesion structures based on volume data, and generate corresponding multi-label segmentation masks; The three-dimensional reconstruction unit is used to extract the three-dimensional mesh surface of each anatomical structure based on multi-label segmentation mask, and to generate triangular meshes using the marching cubes algorithm. Based on the triangular meshes, smoothing and simplification processing is performed to obtain a three-dimensional digital twin model of the target patient. The surgical planning unit is used to set the resection safety boundary and perform three-dimensional expansion with the multi-label segmentation mask, calculate the resection volume, estimate blood loss, analyze the minimum distance between the tumor and blood vessels / bile ducts, and generate a surgical planning report based on the minimum distance to form safety distance information. Physical prior units are used to set biomechanical parameters based on organ and tissue types.

[0032] Furthermore, the medical image loading unit is used to acquire CT or MRI data of the target patient as volume data, supports DICOM sequences and NIfTI format, and can extract metadata such as voxel size, image orientation and patient position based on the volume data.

[0033] Furthermore, the automatic segmentation unit segments relevant organ structures and lesion structures from volume data by using threshold segmentation, region growing, or pre-trained medical image segmentation models, and generates corresponding multi-label segmentation masks. The relevant organ structures include: liver parenchyma, hepatic vein / hepatic artery / portal vein vascular tree and bile duct structure, and the lesion structure is tumor lesion.

[0034] Furthermore, the surgical planning unit performs three-dimensional expansion on the tumor lesion segmentation mask based on the set resection safety boundary (default 10 mm), calculates the resection area volume, estimates blood loss, analyzes the minimum distance between the tumor and blood vessels / bile ducts, assesses the risk level of blood vessel damage (low / medium / high), and generates a surgical planning report containing information such as the tumor center location, resection boundary, and safe distance between blood vessels / bile ducts.

[0035] Furthermore, the biomechanical parameters are: Young's modulus (default 3000 Pascal), Poisson's ratio (default 0.45), density, and damping coefficient, which are used to provide patient-specific material parameters for the intraoperative physical constraint deformation model.

[0036] Furthermore, the preoperative 3D digital twin model, surgical planning report, and biomechanical parameters are packaged into a JSON-formatted surgical planning data package, which is then distributed as global prior knowledge to the following processing modules.

[0037] The neuromorphic perception module is used to acquire raw endoscopic video frames and extract visual feature vectors and raw pulse sequences, and obtain the displacement of anatomical landmarks based on the raw pulse sequences.

[0038] Furthermore, the neuromorphic perception module includes: The visual feature acquisition unit is used to acquire the original endoscopic video frames and input them into the pulse backbone neural network of the first pulse time encoding to obtain the visual feature vector. The raw pulse acquisition unit is used to obtain the raw pulse sequence based on the firing time recorded by the leaky integral firing neurons in the pulse backbone neural network. The anatomical displacement determination unit is used to obtain a pulse feature map based on the original pulse sequence, track key anatomical structures across multiple frames based on the pulse feature map, and obtain the displacement of the anatomical landmarks according to the location of the landmarks in the previous frame and the location of the maximum response point in the current frame.

[0039] Furthermore, the original endoscopic video frames are acquired and processed to obtain normalized 224×224×3 RGB video frames as input images, which are then fed into the Time-to-First-Spike (TTFS) spike backbone neural network for processing. like Figure 2 As shown, the input image first passes through a 7×7 pulsed convolutional layer with a stride of 2, downsampling the spatial size from 224×224 to 112×112, while increasing the number of channels to 64. Then, it passes through a pulsed max pooling layer, further reducing the size to 56×56×64. The data then enters a backbone network consisting of four pulse residual blocks for deep feature extraction: Pulse Residual Block (PR Block 1): Processes and outputs a feature map with a size of 56×56×64; Pulse Residual Block (PR Block 2): Processes and outputs a feature map with a size of 28×28×128; Pulse Residual Block (PR Block 3): Processes and outputs a feature map with a size of 14×14×256; Pulse Residual Block (PR Block 4): Processes and outputs a feature map with a size of 7×7×512; Finally, the feature maps are aggregated into a 512-dimensional visual feature vector after passing through a global average pooling layer.

[0040] Furthermore, the spiking residual block contains two spiking convolutional layers, each followed by a batch normalization layer and a leaky integral firing (LIF) neuron. Skip connections are used to mitigate the vanishing gradient problem during deep network training. The spiking neuron employs the leaky integral firing (LIF) model, and the membrane potential dynamics equation is: ; in, u Represents membrane potential, which is the potential difference across the neuron's cell membrane and is a state variable of the neuron; t Indicates time; t m The membrane time constant determines the membrane potential. u Attenuation (leakage) to resting potential u rest The speed; u rest represents the resting potential, which is the stable membrane potential of a neuron when it is not stimulated; R represents the membrane resistance. I ( t () indicates the input current, which is generated by weighting the presynaptic pulses of the previous layer of neurons; The specific distribution and reset mechanism is as follows: when the membrane potential... u Exceeding the distribution threshold i At a time (the threshold is preset), the neuron generates an output pulse (fires), followed by a membrane potential change. u Reset to u reset (generally u reset < u rest ).

[0041] Furthermore, regarding the operation of TTFS: the entire forward propagation process is completed within 20 time steps. During this process, a first-pulse timing encoding mechanism is used to encode information such as pixel intensity into the pulse firing time; the larger the pixel value, the earlier the pulse is fired. By recording the firing time (or whether it fired) of each neuron in all LIF neurons, the pulse sequence can be obtained. The pulse sequence is the "neuron firing event log" of the network over 20 time steps.

[0042] Further, anatomical landmark displacement generation: This step, based on the pulse feature map, constructs a lightweight anatomical landmark tracking module (obtained through lightweight tracking of the feature map) to track key anatomical structures that are stable across multiple frames (such as blood vessel bifurcation points, liver boundary points, etc.). Let p be the position of the landmark recorded in the previous frame in the feature map. t-1 p is obtained in the current frame by correlation matching or by locating the maximum response point of the impulse spatiotemporal features. t The displacement of the anatomical location is defined as: Δp = p t -p t-1 .

[0043] Furthermore, since the pulse firing event itself is nondifferentiable, this hinders training using the backpropagation algorithm. To address this issue, this invention employs a surrogate gradient function to approximate the gradient of the pulse firing function during training. The SuperSpike surrogate gradient or the fast Sigmoid function is used to approximate the gradient, enabling the spiking neural network to be trained via backpropagation. Sparsity regularization adds a pulse firing rate penalty term to the loss function, encouraging the network to complete feature extraction with a neuron activation rate of no more than 10%, thereby reducing computational cost and energy consumption. The knowledge distillation strategy distills knowledge from the pre-trained traditional CNN teacher model, enabling the student spiking neural network to match the activation pattern of the teacher model in the pulse time domain, thereby improving training efficiency in small sample scenarios. This layer outputs a 512-dimensional visual feature vector containing semantic information of blood vessels, tumors, and anatomical landmarks, with a processing latency of less than 10 milliseconds and a pulse sparsity of less than 0.1.

[0044] Furthermore, the neuromorphic perception module ultimately outputs a 512-dimensional visual feature vector (containing semantic information of blood vessels, tumors, and anatomical landmarks), a pulse sequence, and displacement of anatomical landmark points.

[0045] The pulse sequence and the final extracted 512-dimensional visual feature vector originate from the same forward propagation process, but they express the same visual information in the form of events and continuous features, respectively. Therefore, they are related but independent of each other.

[0046] The liquid adaptive module is used to obtain the deviation intensity of multiple domains based on visual feature vectors.

[0047] Furthermore, the liquid adaptive module includes: The offset intensity determination unit is used to detect various domain offset intensities by inputting data feature vectors into a time-varying weighted liquid neural network and using a domain offset detection controller. The dynamic parameter tuning unit is used to dynamically adjust the confidence threshold, measurement noise covariance, and convergence tolerance in the pulse-assisted optimization module and the physical constraint compensation module based on the domain deviation intensity, so as to obtain the adjustment parameters. For example, when the smoke concentration is detected to be greater than 0.5, the measurement noise covariance of the Kalman filter is amplified by 2 times.

[0048] Furthermore, this invention employs a time-varying weighted liquid neural network to achieve online domain adaptation in surgical scenarios. The domain offset detection controller comprises four liquid neuron units that detect four types of domain offset: smoke, bleeding, changes in illumination, and tissue occlusion. The weight matrix of the liquid neurons changes continuously over time, and the dynamic equation is as follows: ; in, α Indicates the attenuation coefficient. β Indicates fitness rate. w ij This represents the connection weights from neuron j to neuron i in the weight matrix of the liquid neuron. h i and h j Let i and j represent the hidden states of neurons i and j, respectively. s ( h i )express h i The output after activation function σ s ( h j )express h j The output after activation function σ; Closed continuous-time units (CfC) calculate neuron state evolution through ordinary differential equation (ODE) solvers, avoiding discrete time step errors and improving response speed to sudden signals.

[0049] Furthermore, the liquid adaptive module also includes: a feature adaptive normalization unit; The feature adaptive normalization unit is used to maintain the exponential moving average of feature statistics. Based on the deviation between the current frame features and historical statistics, it dynamically adjusts the mean and variance parameters of the batch normalization layer to achieve online feature alignment. This layer does not need to stop and retrain after detecting domain offset, and can complete online adaptation within 20 frames (about 0.67 seconds), with a processing latency of less than 10 milliseconds.

[0050] like Figure 3 As shown, the pulse-assisted optimization module is used to acquire multi-source sensor data during surgery and obtain a measurement pulse sequence. Based on the domain deviation intensity and the state vector of the previous moment, the predicted state at the current moment is obtained, and the camera-organ relative pose is obtained based on the predicted state, the original pulse sequence, and the measurement pulse sequence.

[0051] Furthermore, the pulse-assisted optimization module includes: A parameter conversion unit is used to convert parameters based on domain deviation intensity into fusion adjustment parameters. The data acquisition unit is used to acquire visual measurement data, IMU measurement data, and encoder measurement data during surgery as multi-source sensor data. The prediction output unit is used to input the state vector and fusion adjustment parameters from the previous moment into the constant velocity motion model to obtain the predicted state at the current moment. A multi-source data encoding unit is used to encode multi-source sensor data to obtain a measurement pulse sequence; The gain matrix acquisition unit is used to obtain a continuous value gain matrix by concatenating the pulse sequence based on the measured pulse sequence, the original pulse sequence, and the pulse sequence of the predicted state and inputting them into the pulse gain network. The state update unit is used to update the predicted state and the measurement values ​​corresponding to the multi-source sensor data based on the continuous value gain matrix, remove outliers with Mahalanobis distance exceeding the threshold, and obtain the camera-organ relative pose.

[0052] Furthermore, for the raw pulse sequence from the neuromorphic perception module, since this pulse sequence consists of neuronal firing events generated by the pulsed convolutional backbone network over 20 time steps, it preserves the sparse coding form of visual features evolving over time. In this module, it will be directly input into the pulsed gain network to avoid redundant encoding / quantization of visual features. Simultaneously, the sparsity of the pulses reflects the stability of scene textures and can serve as additional evidence for the reliability of visual measurements.

[0053] Furthermore, the domain offset intensity (smoke, bleeding, sudden changes in illumination, occlusion) output by the liquid adaptive module is converted into parameters such as the measurement noise adjustment coefficient, which are used as fusion adjustment parameters. The pulse-assisted optimization module dynamically adjusts the fusion strategy according to the fusion adjustment parameters, making the filter more stable in complex scenes.

[0054] Furthermore, the multi-source sensor data includes: visual measurement data, IMU measurement data, and encoder measurement data; Among them, visual measurement data: the relative position of the camera is estimated by feature matching or direct visual odometry, which is extracted from the spiking neural network in the neuromorphic perception module. IMU measurement data: triaxial acceleration and angular velocity acquired by the inertial measurement unit; Encoder measurement data: End-effector position provided by the joint angle encoder of the endoscope robotic arm.

[0055] Furthermore, the multi-source data fusion unit uses a pulse gain network to learn the optimal Kalman gain matrix, and combines it with fusion adjustment parameters to fuse multi-source sensor data to obtain fused data x: ; in, p =( p x , p y , p z This indicates the three-dimensional position of the camera relative to the organ, obtained from visual measurement data; q =( q w , q x , q y , q z ) represents the quaternion pose, obtained from visual measurement and encoder measurement data; v =( v x , v y , v z () represents linear velocity. oh =( oh x , oh y , oh z ) represents angular velocity, which is obtained and updated from IMU measurement data.

[0056] Furthermore, the constant velocity motion model used in the prediction output unit is specifically as follows: ; ; Where, x k|k-1 F represents the predicted state at time k based on the measurement at time k-1. kThe state transition matrix (x) describes how the system transitions from the state at time k-1 to the state at time k. k-1|k-1 This represents the state estimate at time k-1, which incorporates all measurements taken at time k-1. P k|k-1 This represents the prediction of the covariance of the state estimation error at time k, based on the measurement at time k-1. P k-1|k-1 The covariance matrix represents the state estimation error at time k-1. k Q represents the current moment. k The process noise covariance matrix represents the uncertainty of random changes in the system state. Its magnitude is dynamically amplified or reduced by the adjustment parameters output by the liquid adaptive module (e.g., when the smoke domain offset intensity is high, the noise increases accordingly). F represents the state transition matrix.

[0057] Furthermore, to avoid reverse encoding of continuous values ​​into the pulse domain, this invention directly uses the raw pulse sequence output by the neuromorphic perception module to construct the measurement input; simultaneously, the measurement values ​​from the three sensors (vision / IMU / encoder) are synchronously encoded into measurement pulse sequences, allowing them to be processed by the pulse gain network in the same time domain. The pulse gain network input is composed of the following parts: the raw pulse sequence, the pulse sequence of the current prediction state, and the measurement pulse sequence of the three types of sensor measurements.

[0058] Furthermore, the pulse gain network comprises three fully connected pulse layers. Its function is to calculate the adaptive Kalman gain based on the pulse input and automatically adjust the weights of different sensors in different scenarios. The output layer is decoded into a continuous-value gain matrix K. k This replaces the fixed gain formula of the traditional Kalman filter: ; Here, H represents the measurement matrix, which maps the state space to the measurement space and describes how state variables affect the measured values. k express k The measurement matrix at any given time may change over time in a time-varying system; R represents the measurement noise covariance, which is dynamically adjusted by the liquid adaptive module. Visual noise increases with the intensity of smoke / obstruction, IMU noise is automatically adjusted with the intensity of vibration (accurate data still needs to be obtained experimentally), and encoder noise is automatically adjusted with the shaking of the robot end effector. Overall, the system achieves adaptation to changes in the statistical characteristics of sensor noise.

[0059] Furthermore, the continuous-value gain matrix K generated using the pulse gain network is... k Execution status update: ; Where, x k|krepresents the final estimated state at time k, after incorporating all sensor measurements; represents the camera-organ relative pose obtained by fusing all sensor measurements in frame k; z k This represents a sequence of measurement pulses from vision, IMU, and encoder. The pulse gain network ensures that the weights of each measurement source are adaptively adjusted according to the real scene.

[0060] Furthermore, sensor data outlier detection and removal: The Mahalanobis distance of the measurement residual is calculated. When the distance exceeds the 95th quantile of the chi-square distribution, it is judged as an outlier and the sensor data is removed, thus achieving robustness against partial sensor failure. At the same time, the judgment threshold is dynamically scaled by the "domain offset intensity" output by the second layer. For example, when the illumination domain offset intensity becomes stronger, the threshold becomes more stringent, and it is necessary to temporarily reduce the weight of the visual measurement value in the formula. The pulse-assisted optimization module ultimately outputs a six-DOF camera-organ relative pose with errors maintained at: position error less than 2mm, attitude error less than 1 degree, update frequency of 30Hz, and processing delay of less than 5 milliseconds.

[0061] The physical constraint compensation module is used to obtain the transformed displacement based on the relative pose of the camera and organ and the displacement of anatomical landmarks. Based on the three-dimensional digital twin model, the transformed displacement and biomechanical parameters, the total loss function is constructed and optimized by the variable field neural network to obtain the three-dimensional mesh model.

[0062] Furthermore, the physical constraint compensation module includes: The coordinate system transformation unit is used to transform the displacement of anatomical landmarks from the camera coordinate system back to the organ coordinate system based on the camera-organ relative pose, and obtain the transformed displacement. The displacement vector acquisition unit is used to input the vertex coordinates of the three-dimensional digital twin model into the deformation field neural network to obtain the three-dimensional displacement vector; The function parameter acquisition unit is used to construct a hyperelastic material model based on biomechanical parameters, obtain the strain energy density function, and obtain the Cauchy stress tensor based on the strain energy density function. Loss function building unit, used to construct a total loss function including multiple boundary condition constraints based on transformed displacement and Cauchy stress tensor; The model optimization unit is used to optimize the deformable field neural network based on the total loss function through an optimizer to obtain the deformed 3D digital twin model and the 3D mesh model.

[0063] Furthermore, the physical constraint compensation module is used to predict and compensate for liver soft tissue deformation caused by breathing, instrument contact, etc. in real time. It receives the six-DOF camera-organ relative pose from the pulse-assisted optimization module, the displacement of anatomical landmarks extracted and tracked in the current frame by the neuromorphic perception module (as boundary conditions), and the three-dimensional digital twin model generated by the preoperative planning system and the patient-specific biomechanical parameters to construct a three-dimensional deformation field that meets biomechanical constraints, and outputs a deformed three-dimensional mesh model consistent with the current frame.

[0064] Furthermore, a fusion and unification operation is required for each input data and coordinate. The six-DOF camera-organ relative pose output by the pulse-assisted optimization module is used for mapping between the current frame image coordinate system and the organ coordinate system. Based on the six-DOF camera-organ relative pose, the continuous anatomical landmark displacements of the neuromorphic perception module are transformed from the camera coordinate system back to the organ coordinate system, ensuring that all subsequent physical solutions are performed in a space consistent with the preoperative digital twin mesh, thus obtaining the transformed displacement.

[0065] Furthermore, after aligning the organ coordinate systems, the first step is to construct the input to the deformable field neural network. The input consists of the coordinates of all vertices of the preoperative 3D digital twin model, representing the 3D spatial coordinates x = ( ) of any point within the liver. x , y , z The deformable field neural network is a multilayer perceptron (MLP) structure used to predict the 3D displacement of each point in a 3D digital twin model. Its structure is as follows: Input layer: Accepts three-dimensional spatial coordinates x = ( x , y , z ); Hidden layers: 4 layers in total, 128 neurons in each layer, with tanh activation function; Output layer: Outputs a three-dimensional displacement vector , which is the displacement vector of the vertex due to deformation.

[0066] Furthermore, to embed physical laws into neural networks, a Neo-Hookean hyperelastic material model is used to describe the mechanical behavior of liver tissue based on biomechanical parameters. The strain energy density function of the Neo-Hookean hyperelastic material model is: ; in, I 1 = tr ( C ) represents the first invariant. tr The trace of a matrix is ​​the sum of the elements along its main diagonal. CThe right Cauchy-Green deformation tensor is a fundamental quantity describing the deformation state of a material. m and l Both represent Lamé constants. m The patient-specific Young's modulus and Poisson's ratio from the biomechanical parameters provided by the preoperative planning module are converted to: , E Indicates Young's modulus. v Represents Poisson's ratio. l From the formula Obtain this strain energy density function, which forms the basis for constructing the physical loss function.

[0067] Furthermore, based on the aforementioned Neo-Hookean hyperelastic material model, the forced deformation field satisfies the force equilibrium equation. ,in, s For the strain energy density function W The derived Cauchy stress tensor describes the forces acting per unit area within a material, where f is the volume force vector, including gravity and external pressure. Here, physical residuals are used, measured by integration over the computational domain Ω, to assess the extent to which the deformation field u deviates from true physical laws. Specifically: ; in, Ω represents the physical residual loss function, used to measure the extent to which the deformation field u satisfies the force balance equation; Ω represents the computational domain of the liver tissue, i.e., the three-dimensional spatial region occupied by the entire digital twin model of the liver; u represents the displacement field vector; and dΩ represents the volume element. The physical residual is used here to ensure that the deformation field predicted by the network is not only accurate at the data points, but also approximately satisfies the basic laws of mechanical equilibrium at every point in space, thereby generating physically reliable deformation results.

[0068] Dirichlet Boundary condition loss : This is used to ensure that the deformation field predicted by the network matches the actual observed displacement at these key landmarks, specifically: ; in, N D express Dirichlet The total number of boundary points, x i Indicates the first i The spatial coordinates of anatomical landmarks with known displacements are determined through visual perception by the neuromorphic perception module. Indicates the first iThe observed displacement vector of a boundary point, that is, the actual amount of movement of that point from the previous frame to the current frame.

[0069] Neumann Boundary condition loss : Used to apply zero-traction boundary conditions to the surface of organs to simulate free surfaces, specifically: ; in, This represents the boundary surface to which Neumann boundary conditions have been applied, where n represents the boundary surface. The unit outward normal vector on, This represents the traction force vector, the projection of the stress tensor onto the normal direction, i.e., the external force per unit area of ​​the surface. Representing an area element, the whole is achieved through... To force the traction force to be 0.

[0070] Contact boundary conditions: To prevent deformed tissue from penetrating the surface of the surgical instrument, inequality constraints are applied.

[0071] H 1 Regularization loss A penalty is applied to the first derivative of the deformable field (displacement gradient) to improve the smoothness and stability of the solution, specifically: ; in, The displacement gradient tensor is a second-order tensor (3×3 matrix) that describes the rate of change of the displacement field in space and contains strain information. H represents the Frobenius norm, which is the square root of the sum of the squares of all elements of the matrix; 1 Regularization loss By penalizing excessive displacement gradients, non-physical high-frequency oscillations and drastic changes in the deformation field can be effectively suppressed.

[0072] Furthermore, the total loss function Taking into account all the above constraints and physical residuals, a weighted summation is used to balance data fitting, physical laws, boundary conditions, and smoothness requirements, specifically: ; in, These are the weighting coefficients for each loss, used to balance the importance of different constraints. They need to be adjusted in practice to achieve the best balance.

[0073] The transformed anatomical landmark displacements are converted into Dirichlet boundary conditions for the model, requiring the network to output displacements at these points that are consistent with actual observations. Neumann boundary conditions are applied to the free surface of the 3D digital twin model (i.e., the area not in contact with the instrument) to simulate zero traction on the free surface of the tissue.

[0074] Furthermore, after incorporating all input data into the physical constraints, the model optimization unit optimizes the deformation field neural network within a finite number of steps using the L-BFGS optimizer based on the total loss function to obtain the continuous 3D deformation field corresponding to the current frame. After optimization, the obtained deformation field u(x) is applied to the vertex coordinates of the preoperative 3D digital twin model to generate the deformed digital twin model. : ; in, This represents the original coordinates of the vertices in the preoperative 3D mesh model, i.e., their position coordinates before deformation; Based on the deformed digital twin model A deformed 3D mesh model consistent with the actual organ morphology in the current frame is obtained, with a target registration error of less than 1.5 mm, an update frequency of 10 Hz (updates once every 3 frames for 30 fps video), and a processing latency of less than 15 milliseconds.

[0075] The augmented reality visualization module is used to obtain a two-dimensional mask based on the relative pose of the camera and organ and the three-dimensional mesh model. It assigns corresponding colors to different anatomical structures and combines them with the two-dimensional mask to obtain a colored mesh mask. This mask is then superimposed on the original endoscopic video frame to output an AR superimposed video frame. Real-time distance warnings are provided based on the surgical planning report.

[0076] Furthermore, the augmented reality visualization module includes: The camera projection unit is used to construct a projection matrix based on the relative pose of the camera and organ, and to project the vertices of the 3D mesh model onto the 2D image plane based on the projection matrix and the camera intrinsic parameter matrix. Mesh rasterization unit is used to rasterize triangular meshes based on a two-dimensional image plane using a depth buffer algorithm to process occlusion relationships and generate a two-dimensional mask for the organ surface. Anatomical structure color matching unit, used to assign a unique corresponding color to different anatomical structures to obtain a color matching scheme; The coloring unit is used to color the two-dimensional mask based on the color scheme to obtain a colored mesh mask; An overlay display unit is used to overlay the original endoscope video frame with a shading mesh mask to output an AR overlay video frame. The safety distance warning unit is used to extract the safe boundary distance for tumor resection, the minimum safe distance for major blood vessels, and the minimum safe distance for bile ducts from the surgical planning report as safety thresholds. It also acquires the Euclidean distance between the tip of the surgical instrument and the tumor boundary, the surface of the blood vessel, and the bile duct in real time. When the distance is less than the corresponding safety threshold, a warning is issued.

[0077] Furthermore, a projection matrix is ​​constructed based on the camera-organ relative pose: ; in, R The rotation matrix is ​​a 3×3 matrix that describes the rotation transformation of the organ coordinate system relative to the camera coordinate system. t The translation vector is a 3×1 vector that describes the translation of the origin of the organ coordinate system relative to the origin of the camera coordinate system. It is used directly as an extrinsic parameter of the projection matrix and combined with the camera intrinsic parameter matrix. K Multiplying (the camera intrinsic parameter matrix is ​​the camera's inherent parameters) projects the vertices of the 3D mesh onto the 2D image plane: ; in, p 3D Represents the vertices of a 3D mesh. The 3D coordinates of the vertices in the 3D mesh model are represented in organ coordinate system.

[0078] Furthermore, the anatomical structure color matching unit is used to assign a unique corresponding color to different anatomical structures, resulting in a color scheme: The liver parenchyma was brownish-red, with rgba values ​​of 0.5, 0.3, 0.2, and 0.5. The tumor was yellow, and the rgba values ​​were 1.0, 1.0, 0.0, and 0.7. The blood vessels are red; RGBA values ​​are 1.0, 0.0, 0.0, and 0.8. The bile duct is green; RGBA values ​​are 0.0, 1.0, 0.0, and 0.8. Here, rgba represents the color space of Red, Green, Blue, and Alpha.

[0079] Furthermore, the overlay display unit is used to perform Alpha channel mixing and overlay based on the shading mesh mask and the original endoscopic video frame, and output AR overlay video frame. I output :

[0080] in, α This represents the transparency parameter, with a default value of 0.5. I overlayThis represents the shaded mesh mask image, which is the visualization of the rendered 3D model on a 2D plane. I video This represents the original endoscopic video frame.

[0081] Furthermore, safety distance information is extracted from the surgical planning report as safety thresholds, including: safe distance for tumor resection (10 mm), minimum safe distance threshold for major blood vessels (hepatic vein, hepatic artery, portal vein) (default 5 mm), and minimum safe distance threshold for bile duct (default 10 mm). The safe distance warning unit calculates the Euclidean distance between the tip of the surgical instrument (identified by color or shape detection) and the tumor boundary, blood vessel surface, and bile duct in real time. When the distance is less than the safe threshold (10mm for the tumor boundary, 5mm for the blood vessel, and 10mm for the bile duct), it is highlighted with a red warning mark on the superimposed image and triggers an audible or tactile alarm.

[0082] Furthermore, the augmented reality visualization module also includes: Head-up display unit, used to overlay key performance indicators on AR overlaid video frames; Key performance metrics include: real-time frame rate (FPS), latency of each module (SNN, Kalman, PINN, Vis), total latency, domain offset intensity (0-1 normalized values ​​of smoke, bleed, illumination, and occlusion), state estimation confidence, and current minimum safe distance.

[0083] Furthermore, the augmented reality visual module outputs video frames with AR overlay, and thanks to the high speed and low latency characteristics of 5G technology, the display latency is controlled at the millisecond level.

[0084] Furthermore, when the liquid adaptive module detects a high-intensity domain offset or the confidence level of the pulse-assisted optimization module is lower than the safety threshold, it automatically reduces the update frequency of the physical constraint compensation module PINN (from 10Hz to 5Hz) or increases the number of optimization steps to balance accuracy and real-time performance.

[0085] Furthermore, the aforementioned modules are executed asynchronously in independent threads or GPU streams, and data is passed through a circular buffer to reduce synchronization waiting time.

[0086] Furthermore, it also includes: a failure safety monitoring module; The failure safety monitoring module is used to monitor and determine in real time whether the outputs of the neuromorphic perception module, the liquid adaptive module, the pulse-assisted optimization module, and the physical constraint compensation module are abnormal. like Figure 4As shown, when the neuromorphic perception module outputs abnormalities: impulse sparsity > 0.5 (indicating that the network has degenerated into dense activation) or the variance of the output feature vector is abnormal, the model is switched to MobileNet or EfficientNet for feature extraction. When the liquid adaptive module outputs an abnormal value: domain offset intensity > 0.9 (indicating that the scene exceeds the training distribution) or the confidence modulation module outputs an abnormally negative value, the adaptive parameters are frozen and the system reverts to the initial fixed parameters. When the pulse-assisted optimization module outputs an abnormality, such as the Kalman gain not satisfying positive definiteness or generating numerical overflow, it automatically switches to the classic Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF), using a fixed gain calculation formula. When the physical constraint compensation module outputs an abnormality: the physical residual exceeds the convergence tolerance or the optimization iteration fails to converge, it will revert to rigid registration (only translation and rotation, without deformation) or use the pre-calculated average deformation template. Each module output includes a confidence score, and a warning is displayed when the global confidence score is less than 0.5.

[0087] Furthermore, in addition to numerical results, each of the above modules outputs a confidence score. Downstream modules dynamically adjust their fault tolerance thresholds and weight allocations based on the upstream confidence score. When the global confidence score is lower than the safety threshold of 0.5, a "low confidence" warning is displayed on the HUD, and the surgeon is advised to switch to traditional navigation mode or direct vision surgery.

[0088] Furthermore, it also includes a hardware deployment module: By deploying GPU acceleration modules on NVIDIA GPUs and leveraging the CUDA parallel computing architecture, we are able to significantly improve the efficiency of tensor operations and physical residual calculations in spiking neural networks, PINN, and the rendering module. The neuromorphic hardware interface supports exporting spiking neural networks to ONNX format or Loihi 2 proprietary format, which can be deployed on the Intel Loihi 2 neuromorphic chip to achieve ultra-low power (<1 watt) execution of the perception layer. Docker containerization deployment packages the preoperative planning subsystem and the intraoperative navigation subsystem into independent Docker images, supporting flexible deployment between edge computing devices, mobile operating vehicles, and cloud servers; The ROS2 robot integration interface communicates with robotic surgical platforms (such as the da Vinci surgical robot) through the ROS2 middleware, subscribes to topics on the joint status of the robotic arm, and publishes AR overlay images and safety alarm topics to achieve closed-loop control.

[0089] Example 2 Based on the same inventive concept, embodiments of the present invention also provide a real-time surgical navigation method based on neuromorphic computing and physical constraints, including: Obtain the target patient's body data to generate a multi-label segmentation mask, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters; Original endoscopic video frames were acquired and visual feature vectors and original pulse sequences were extracted. The displacement of anatomical landmarks was obtained based on the original pulse sequences. Multiple domain deviation intensities are obtained based on visual feature vectors; The system acquires multi-source sensor data during surgery and obtains a measurement pulse sequence. Based on the domain deviation intensity and the state vector of the previous moment, the predicted state at the current moment is obtained. Based on the predicted state, the original pulse sequence, and the measurement pulse sequence, the camera-organ relative pose is obtained. The transformed displacement is obtained based on the relative pose of the camera and organ and the displacement of anatomical landmarks. The total loss function is constructed based on the three-dimensional digital twin model, the transformed displacement and biomechanical parameters, and the variable field neural network is optimized to obtain a three-dimensional mesh model. A two-dimensional mask is obtained based on the relative pose of the camera and organ and the three-dimensional mesh model. Different colors are assigned to different anatomical structures and combined with the two-dimensional mask to obtain a colored mesh mask. This mask is then superimposed on the original endoscopic video frame to output an AR superimposed video frame. Real-time distance warnings are provided based on the surgical planning report.

[0090] Furthermore, the implementation process of each step in this embodiment corresponds one-to-one with the implementation process of the above-mentioned functional modules, and will not be described in detail here.

[0091] Example 3 Based on the same inventive concept, the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores instructions, characterized in that the instructions are loaded and executed by the processor to implement the real-time surgical navigation method based on neuromorphic computing and physical constraints as in Embodiment 2.

[0092] Based on the same inventive concept, the present invention also provides a computer device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When the processor executes the program stored in the memory, it can implement the real-time surgical navigation method based on neuromorphic computing and physical constraints as in Example 2.

[0093] The electronic device may include a processor, a communications interface, a memory, and a communication bus, wherein the processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute the real-time surgical navigation method based on neuromorphic computing and physical constraints described in Embodiment 2.

[0094] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0095] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0096] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A real-time surgical navigation system based on neuromorphic computing and physical constraints, characterized in that, include: The preoperative planning module is used to acquire the target patient's body data, generate multi-label segmentation masks, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters; The neuromorphic perception module is used to acquire raw endoscopic video frames and extract visual feature vectors and raw pulse sequences, and obtain the displacement of anatomical landmarks based on the raw pulse sequences. A liquid adaptive module is used to obtain the deviation intensity of multiple domains based on the visual feature vector; The pulse-assisted optimization module is used to acquire multi-source sensor data during surgery and obtain a measurement pulse sequence. Based on the domain deviation intensity and the state vector at the previous moment, the predicted state at the current moment is obtained, and the camera-organ relative pose is obtained based on the predicted state, the original pulse sequence, and the measurement pulse sequence. The physical constraint compensation module is used to obtain the transformed displacement based on the relative pose of the camera and organ and the displacement of the anatomical landmarks, construct the total loss function based on the three-dimensional digital twin model, the transformed displacement and the biomechanical parameters, and optimize the variable field neural network to obtain a three-dimensional mesh model. The augmented reality visualization module is used to obtain a two-dimensional mask based on the camera-organ relative pose and the three-dimensional mesh model, assign corresponding colors to different anatomical structures and combine them with the two-dimensional mask to obtain a colored mesh mask, and superimpose it with the original endoscopic video frame to output an AR superimposed video frame, and provide real-time distance warning based on the surgical planning report.

2. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 1, characterized in that, The preoperative planning module includes: A medical image loading unit is used to acquire CT or MRI data of the target patient as the volume data; An automatic segmentation unit is used to segment related organ structures and lesion structures based on the volume data, and generate the corresponding multi-label segmentation mask; The three-dimensional reconstruction unit is used to extract the three-dimensional mesh surface of each anatomical structure based on the multi-label segmentation mask, generate a triangular mesh, and perform smoothing and simplification processing based on the triangular mesh to obtain the three-dimensional digital twin model of the target patient. The surgical planning unit is used to set the resection safety boundary and perform three-dimensional expansion with the multi-label segmentation mask, calculate the resection volume, estimate blood loss, analyze the minimum distance between the tumor and blood vessels / bile ducts, and generate safety distance information based on the minimum distance to form the surgical planning report. A physical prior unit is used to set the biomechanical parameters according to the organ and tissue type.

3. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 2, characterized in that, The neuromorphic perception module includes: A visual feature acquisition unit is used to acquire the original endoscopic video frame and input it into the pulse backbone neural network of the first pulse time encoding to obtain the visual feature vector. The original pulse acquisition unit is used to obtain the original pulse sequence based on the firing time recorded by the leaky integral firing neurons in the pulse backbone neural network; The anatomical displacement determination unit is used to obtain a pulse feature map based on the original pulse sequence, track key anatomical structures across multiple frames based on the pulse feature map, and obtain the displacement of the anatomical landmarks according to the location of the landmarks in the previous frame and the location of the maximum response point in the current frame.

4. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 3, characterized in that, The liquid adaptive module includes: The offset intensity determination unit is used to detect various domain offset intensities by inputting the data feature vector into a time-varying weighted liquid neural network and using a domain offset detection controller. The dynamic parameter tuning unit is used to dynamically adjust the confidence threshold, measurement noise covariance, and convergence tolerance in the pulse-assisted optimization module and the physical constraint compensation module based on the domain deviation intensity, so as to obtain the adjustment parameters.

5. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 4, characterized in that, The pulse-assisted optimization module includes: The parameter conversion unit is used to convert the domain deviation intensity into fusion adjustment parameters; The data acquisition unit is used to acquire visual measurement data, IMU measurement data, and encoder measurement data during surgery as the multi-source sensor data. The prediction output unit is used to input the state vector of the previous moment and the fusion adjustment parameters into the constant velocity motion model to obtain the predicted state at the current moment. A multi-source data encoding unit is used to encode the multi-source sensor data to obtain the measurement pulse sequence; The gain matrix acquisition unit is used to input the pulse gain network after splicing the measured pulse sequence, the original pulse sequence, and the pulse sequence of the predicted state to obtain a continuous value gain matrix; The state update unit is used to update the predicted state and the measurement values ​​corresponding to the multi-source sensor data based on the continuous value gain matrix, remove outliers with Mahalanobis distance exceeding the threshold, and obtain the camera-organ relative pose.

6. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 5, characterized in that, The physical constraint compensation module includes: A coordinate system transformation unit is used to transform the displacement of the anatomical landmark from the camera coordinate system back to the organ coordinate system based on the camera-organ relative pose, so as to obtain the transformed displacement. The displacement vector acquisition unit is used to input the vertex coordinates of the three-dimensional digital twin model into the deformation field neural network to obtain the three-dimensional displacement vector; The function parameter acquisition unit is used to construct a hyperelastic material model based on the biomechanical parameters, obtain the strain energy density function, and obtain the Cauchy stress tensor based on the strain energy density function. The loss function construction unit is used to construct the total loss function, which includes multiple boundary condition constraints, based on the transformed displacement and the Cauchy stress tensor. The model optimization unit is used to optimize the deformable field neural network based on the total loss function through an optimizer to obtain the deformed three-dimensional digital twin model and the three-dimensional mesh model.

7. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 6, characterized in that, The augmented reality visual module includes: A camera projection unit is used to construct a projection matrix based on the camera-organ relative pose, and to project the vertices of the three-dimensional mesh model onto a two-dimensional image plane based on the projection matrix and the camera intrinsic parameter matrix. The grid rasterization unit is used to perform rasterization processing of triangular grids based on the two-dimensional image plane using a depth buffer algorithm to generate a two-dimensional mask of the organ surface. Anatomical structure color matching unit, used to assign a unique corresponding color to different anatomical structures to obtain a color matching scheme; A coloring unit is used to color the two-dimensional mask based on the color scheme to obtain the color mesh mask; An overlay display unit is used to overlay the original endoscope video frame with the shading mesh mask and output the AR overlay video frame. The safety distance warning unit is used to extract the tumor resection safety boundary distance, the minimum safety distance of major blood vessels, and the minimum safety distance of bile duct as safety thresholds based on the surgical planning report, and to obtain the Euclidean distance between the tip of the surgical instrument and the tumor boundary, the surface of the blood vessel, and the bile duct in real time. When the distance is less than the corresponding safety threshold, a warning is issued.

8. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 7, characterized in that, The augmented reality visual module also includes: A head-up display unit is used to overlay and display key performance indicators on the AR overlay video frame; The key performance indicators include: real-time frame rate, latency of each module, total latency, domain offset strength, state estimation confidence, and current minimum safe distance.

9. The real-time surgical navigation system based on neuromorphic computing and physical constraints according to claim 8, characterized in that, Also includes: Failure safety monitoring module; The failure safety monitoring module is used to monitor and determine in real time whether the outputs of the neuromorphic perception module, the liquid adaptive module, the pulse-assisted optimization module, and the physical constraint compensation module are abnormal. When the neuromorphic perception module outputs an abnormality: the impulse sparsity is >0.5 or the variance of the output feature vector is abnormal, the module switches to the MobileNet or EfficientNet model for feature extraction. When the liquid adaptive module outputs an abnormal value (domain offset > 0.9) or the confidence modulation module outputs an abnormal negative value, the adaptive parameters are frozen and the system reverts to the initialized fixed parameters. When the pulse-assisted optimization module outputs an abnormality: the Kalman gain does not meet the positive definiteness or generates a numerical overflow, it automatically switches to the classic extended Kalman filter (EKF) or unscented Kalman filter (UKF) and uses a fixed gain calculation formula. When the physical constraint compensation module outputs an abnormality: the physical residual exceeds the convergence tolerance or the optimization iteration fails to converge, it will revert to rigid registration or use the pre-calculated average deformation template. Each module output includes a confidence score, and a warning is displayed when the global confidence score is less than 0.

5.

10. A real-time surgical navigation method based on neuromorphic computation and physical constraints, applied to the real-time surgical navigation system based on neuromorphic computation and physical constraints as described in any one of claims 1-9, characterized in that, include: Obtain the target patient's body data to generate a multi-label segmentation mask, and generate a three-dimensional digital twin model and surgical planning report, and set biomechanical parameters; The original endoscopic video frames are acquired and visual feature vectors and original pulse sequences are extracted. The displacement of anatomical landmarks is obtained based on the original pulse sequences. Multiple domain deviation intensities are obtained based on the aforementioned visual feature vectors; Acquire multi-source sensor data during surgery and obtain a measurement pulse sequence. Based on the domain deviation intensity and the state vector at the previous moment, the predicted state at the current moment is obtained. Based on the predicted state, the original pulse sequence, and the measurement pulse sequence, the camera-organ relative pose is obtained. The transformed displacement is obtained based on the relative pose of the camera and organ and the displacement of the anatomical landmarks. The total loss function is constructed based on the three-dimensional digital twin model, the transformed displacement and the biomechanical parameters, and the variable field neural network is optimized to obtain a three-dimensional mesh model. A two-dimensional mask is obtained based on the camera-organ relative pose and the three-dimensional mesh model. A corresponding color is assigned to the two-dimensional mask based on different anatomical structures to obtain a colored mesh mask. This mask is then superimposed on the original endoscopic video frame to output an AR superimposed video frame. Real-time distance warnings are provided based on the surgical planning report.

Images