Augmented reality intelligent navigation and operation quantitative evaluation method and system in operating room

Abstract

The invention discloses an augmented reality intelligent navigation and operation quantitative evaluation method and system in an operating room, and the method comprises the steps: collecting multispectral image data and depth point cloud data of an operation region in the operating room in real time, and generating a multidimensional perception data set; performing feature extraction and spatial registration on the multi-dimensional perception data set, and decomposing the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer; performing dynamic matching and real-time rendering on the anatomical structure feature layer and the instrument interaction feature layer based on a digital twinning technology to generate augmented reality navigation information and an operation guidance scheme; and performing multi-dimensional quantitative analysis on the surgical operation process according to the augmented reality navigation information and the operation guidance scheme, and outputting a quantitative evaluation result. By utilizing the embodiment of the invention, augmented reality navigation guidance which is synchronous with a real operation scene in real time can be provided, meanwhile, accurate quantitative evaluation is carried out on the operation operation, and the accuracy, the safety and the trainability of the operation are improved.

Description

Technical Field

[0001] This invention belongs to the field of medical technology, specifically an augmented reality intelligent navigation and operation quantitative evaluation method and system for operating rooms. Background Technology

[0002] In the current field of surgery, traditional image-guided navigation systems primarily rely on preoperative 2D images such as CT and MRI for 3D reconstruction, providing surgeons with anatomical references. However, these systems have significant limitations: First, they cannot perceive and reflect dynamic anatomical changes during surgery caused by tissue displacement, deformation, or instrument manipulation in real time, resulting in information lag and affecting navigation accuracy. Second, existing systems focus on spatial navigation, lacking the ability to quantitatively monitor and evaluate key indicators of the surgical procedure itself, such as instrument trajectory and manipulation force, making objective evaluation of surgical quality and precise teaching difficult. Furthermore, the application of augmented reality technology in surgery is mostly limited to the overlay display of static models, failing to deeply integrate with the dynamic surgical scene, resulting in weak interactive guidance. Summary of the Invention

[0003] The purpose of this invention is to provide an augmented reality intelligent navigation and operation quantitative evaluation method and system in the operating room to overcome the shortcomings of the prior art. It can provide augmented reality navigation guidance that is synchronized with the real surgical scene in real time, and at the same time perform accurate quantitative evaluation of surgical operations, thereby improving the accuracy, safety and trainability of the surgery.

[0004] One embodiment of this application provides an augmented reality intelligent navigation and operation quantitative evaluation method in an operating room, the method comprising: Real-time acquisition of multispectral image data and depth point cloud data of the surgical area in the operating room, combined with surgical instrument motion trajectory data, generates a multidimensional perception data set of the surgical scene; Feature extraction and spatial registration are performed on the multidimensional perception data set to construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structures, and the dynamic three-dimensional augmented reality model is decomposed into an anatomical structure feature layer and an instrument interaction feature layer. Based on digital twin technology, the anatomical structure feature layer and the instrument interaction feature layer are dynamically matched and rendered in real time to generate augmented reality navigation information and operation guidance scheme. Based on the augmented reality navigation information and operation guidance scheme, a multi-dimensional quantitative analysis of the surgical operation process is performed, and a quantitative evaluation result including operation accuracy and execution efficiency is output.

[0005] Optionally, the real-time acquisition of multispectral image data and depth point cloud data of the surgical area within the operating room, combined with surgical instrument motion trajectory data, generates a multidimensional perception data set of the surgical scene, including: Simultaneously acquire multispectral image sequences and depth point cloud sequences of the surgical area to generate raw multimodal sensing data streams; Real-time capture of the spatial position and attitude changes of surgical instruments, generating surgical instrument motion trajectory data; A timestamp alignment algorithm is used to synchronize the original multimodal sensing data stream with the surgical instrument motion trajectory data in time, eliminating the time delay between acquisition devices and generating a time-synchronized multi-source data stream; Spatial coordinate unification processing is performed on time-synchronized multi-source data streams. Through coordinate transformation, all data are mapped to the same world coordinate system, ultimately generating a multi-dimensional perception data set of the surgical scene.

[0006] Optionally, the step of performing feature extraction and spatial registration on the multidimensional perception data set to construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structures, and decomposing the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer, includes: Tissue features from multispectral images and geometric features from depth point clouds are extracted from multidimensional sensory datasets. Feature descriptor algorithms are used to identify the boundaries of key anatomical structures and generate an initial feature set of anatomical structures. Based on the initial feature set of anatomical structures, the real-time acquired anatomical features are spatially registered with the pre-constructed standard anatomical model through the iterative nearest point algorithm to generate a registered accurate anatomical structure model. By combining surgical instrument motion trajectory data, the relative positional relationship between instruments and tissues is dynamically updated in the registered precise anatomical structure model, and a dynamic three-dimensional augmented reality model of key surgical anatomical structures is constructed. The dynamic 3D augmented reality model is decomposed hierarchically, separating the static anatomical structure feature layer and the dynamic instrument interaction feature layer to generate a dual-feature layer model representation.

[0007] Optionally, the step of dynamically matching and rendering the anatomical structure feature layer and the instrument interaction feature layer based on digital twin technology to generate augmented reality navigation information and operation guidance scheme includes: A digital twin of the surgical scene is created, and the anatomical structure feature layer and the instrument interaction feature layer are imported into the digital twin engine to generate an initial digital twin model; In the initial digital twin model, dynamic matching of the anatomical structure feature layer and the instrument interaction feature layer is realized. The interaction state between the instrument and the anatomical structure is calculated in real time through a collision detection algorithm to generate dynamic matching results. Based on the dynamic matching results, a real-time rendering pipeline is used to visualize the digital twin model, overlaying key anatomical structures and instrument path guidance to generate an augmented reality navigation view. Based on the augmented reality navigation view and surgical procedure knowledge base, an operation guidance plan containing operation steps, risk warnings and alternative solutions is automatically generated, and finally, a complete augmented reality navigation information and operation guidance plan is output.

[0008] Optionally, the step of performing multi-dimensional quantitative analysis of the surgical procedure based on the augmented reality navigation information and operation guidance scheme, and outputting a quantitative evaluation result including operational accuracy and execution efficiency, includes: The actual movement trajectory of instruments during surgical procedures is recorded in real time and compared with the expected path in augmented reality navigation information to generate instrument movement deviation data. Based on the instrument motion deviation data, the instrument positioning accuracy, path tracking error and operational stability index are calculated to generate an operational accuracy evaluation dataset. Based on the time requirements of the operation guidance plan, analyze the difference between the actual time consumption and the standard time consumption of each operation step, calculate the time efficiency and process continuity indicators, and generate an execution efficiency evaluation dataset. By integrating the operational accuracy assessment dataset and the execution efficiency assessment dataset, a comprehensive score is generated through a multi-index fusion algorithm, and the final output is a quantitative assessment result that includes both operational accuracy and execution efficiency.

[0009] Another embodiment of this application provides an augmented reality intelligent navigation and operation quantitative evaluation system in an operating room, the system comprising: The acquisition module is used to acquire multispectral image data and depth point cloud data of the surgical area in the operating room in real time, and combine them with surgical instrument motion trajectory data to generate a multidimensional perception data set of the surgical scene. The extraction module is used to extract features and spatially register the multidimensional perception data set, construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structure, and decompose the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer. The generation module is used to dynamically match and render the anatomical structure feature layer and the instrument interaction feature layer in real time based on digital twin technology, and generate augmented reality navigation information and operation guidance scheme. The evaluation module is used to perform multi-dimensional quantitative analysis of the surgical procedure based on the augmented reality navigation information and operation guidance scheme, and output quantitative evaluation results including operation accuracy and execution efficiency.

[0010] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.

[0011] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.

[0012] Compared with existing technologies, this invention provides an augmented reality intelligent navigation and operation quantitative evaluation method in the operating room. It collects multispectral image data and depth point cloud data of the surgical area in real time to generate a multidimensional perception data set of the surgical scene. Feature extraction and spatial registration are performed on the multidimensional perception data set to construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structures. This dynamic three-dimensional augmented reality model is then decomposed into an anatomical structure feature layer and an instrument interaction feature layer. Based on digital twin technology, the anatomical structure feature layer and the instrument interaction feature layer are dynamically matched and rendered in real time to generate augmented reality navigation information and operation guidance schemes. Based on the augmented reality navigation information and operation guidance schemes, the surgical operation process is quantitatively analyzed in multiple dimensions, and quantitative evaluation results are output. This provides augmented reality navigation guidance that is synchronized with the real surgical scene in real time, while simultaneously providing precise quantitative evaluation of the surgical operation, improving the accuracy, safety, and trainability of the surgery. Attached Figure Description

[0013] Figure 1 A hardware structure block diagram of a computer terminal for an augmented reality intelligent navigation and operation quantification evaluation method in an operating room, provided in an embodiment of the present invention; Figure 2 A flowchart illustrating an augmented reality intelligent navigation and operation quantification evaluation method in an operating room, provided by an embodiment of the present invention; Figure 3 This is a schematic diagram of an augmented reality intelligent navigation and operation quantification evaluation system in an operating room, provided as an embodiment of the present invention. Detailed Implementation

[0014] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0015] The present invention first provides an augmented reality intelligent navigation and operation quantitative evaluation method in the operating room. This method can be applied to electronic devices, such as computer terminals, specifically ordinary computers.

[0016] The following detailed explanation uses a computer terminal as an example. Figure 1 This is a hardware structure block diagram of a computer terminal for an augmented reality intelligent navigation and operation quantification evaluation method in an operating room, provided as an embodiment of the present invention. Figure 1As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0017] The non-volatile storage medium can store the operating system and computer program. The computer program includes program instructions that, when executed, cause the processor to perform any augmented reality intelligent navigation and operational quantitative evaluation method within the operating room.

[0018] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0019] Internal memory provides an environment for the execution of computer programs in non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any augmented reality intelligent navigation and operation quantification evaluation method in the operating room.

[0020] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0021] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0022] See Figure 2 The present invention provides an augmented reality intelligent navigation and operation quantitative evaluation method in an operating room, which may include the following steps: S201 collects multispectral image data and depth point cloud data of the surgical area in the operating room in real time, and combines them with surgical instrument motion trajectory data to generate a multidimensional perception data set of the surgical scene. Specifically, it can simultaneously acquire multispectral image sequences and depth point cloud sequences of the surgical area to generate raw multimodal sensing data streams; This step is fundamental to building surgical scene awareness. It involves acquiring optical and geometric information of the surgical area through collaborative data collection from multiple devices. The specific implementation is as follows: Multispectral image sequence acquisition employed a multispectral camera system containing six optical sensors covering the visible light (400-700nm) and near-infrared (700-1000nm) ranges. Specifically, 450nm (blue light) was used to identify blood vessel distribution, 630nm (red light) to distinguish bleeding areas, and 900nm (near-infrared) to penetrate superficial tissue and observe deep structures. The camera resolution was set to 1920×1080 pixels, with a frame rate of 30fps to ensure capture of dynamic details during the surgical procedure. The lens focal length was 25mm, with a 60° field of view, covering the surgical incision and a 10cm perimeter. Acquisition was synchronously triggered by the surgical light. Each frame was accompanied by a timestamp accurate to milliseconds (e.g., "2025-10-24 14:30:05.123") and saved in RAW format (uncompressed, preserving original spectral information) to avoid loss of image details.

[0023] The depth point cloud sequence acquisition utilizes a structured light depth sensor. Its principle involves projecting an coded grating onto the surgical area and calculating the 3D coordinates of each pixel by analyzing grating distortion. The sensor sampling frequency is 20fps (matching the image sequence frame rate to balance accuracy and data volume), with a point cloud density of 500 points / square centimeter (i.e., 500 3D coordinate points per square centimeter). Single-point positioning accuracy is ±0.1mm, ensuring the capture of subtle undulations on the tissue surface (such as tumor protrusions and blood vessel pathways). Point cloud data is stored in frames, with each frame containing approximately 2 million points (corresponding to a 10×10cm surgical area). Each point's information includes (x, y, z) 3D coordinates, reflectance intensity (reflecting tissue reflectivity, such as higher reflectance in bone than soft tissue), and a timestamp (from the same source as the multispectral image timestamp for easy subsequent synchronization).

[0024] The raw multimodal sensing data stream is a collection of multispectral image sequences and depth point cloud sequences, arranged chronologically. For example, frame 1 contains a multispectral image (6 bands) and a depth point cloud (2 million points) at time 14:30:05.123, and frame 2 contains data at time 14:30:05.156 (interval approximately 33ms, corresponding to a frame rate of 30fps). The data stream needs to be stored in real-time in a local high-speed cache (read / write speed > 1GB / s) to avoid data loss due to storage latency. Simultaneously, an integrity check is performed every 100 frames to ensure no frames are lost or corrupted, providing continuous and complete raw data for subsequent processing.

[0025] Real-time capture of the spatial position and attitude changes of surgical instruments, generating surgical instrument motion trajectory data; This step provides data support for analyzing the interaction between surgical instruments and tissues by precisely tracking the dynamics of the surgical instruments. Specifically, it is implemented as follows: Surgical instrument tracking employs an optical positioning system comprising two infrared cameras (1280×1024 pixels resolution, 100fps), mounted 3 meters above each other on either side of the operating table, forming a binocular visual positioning range covering the entire surgical area (2 meters in diameter). Three infrared reflective markers (3mm in diameter, reflectivity >90%) are fixed to the instrument surface, arranged in an equilateral triangle (2cm side length) to ensure that at least two markers are captured by the camera in any orientation.

[0026] Spatial position capture is achieved through triangulation: an infrared camera captures images of the marker point every 10ms (100fps), extracts the two-dimensional coordinates of the marker point in the image, and combines the intrinsic and extrinsic parameters of the binocular camera (intrinsic parameters include focal length and principal point coordinates, extrinsic parameters include the relative positions of the two cameras) to calculate the three-dimensional world coordinates (x, y, z) of the marker point. The positioning accuracy reaches 0.1mm, and the repeatability error is <0.05mm. For example, the coordinates of a marker point on a hemostat at time t are (50.2, 30.5, 15.8) mm, and at time t+10ms they are (51.0, 30.7, 15.9) mm, reflecting the minute movement of the instrument.

[0027] Attitude change capture is achieved through calculation of the relative positions of marker points: Based on the 3D coordinates of three marker points, a local coordinate system for the instrument is constructed (with one marker point as the origin, and the other two marker points defining the x-axis and xy-plane). By transforming the local coordinate system with the world coordinate system, the instrument's attitude parameters—Eulerian angles (roll, pitch, and yaw)—are solved with an accuracy of ±0.5°. For example, a change in the scalpel's roll angle from 0° to 5° indicates a 5° rotation of the instrument around its own axis, potentially corresponding to an adjustment in the cutting direction.

[0028] Surgical instrument motion trajectory data is a sequence of position and orientation over time, with one data point recorded every 10ms. The format is (timestamp, x, y, z, roll, pitch, yaw), for example, "2025-10-24 14:30:05.123, 50.2,30.5, 15.8, 0.0°, 2.0°, 1.5°". The trajectory data needs to be associated with the instrument type (e.g., distinguishing between scalpels and hemostatic forceps by the distribution pattern of marker points) and transmitted to the data processing unit in real time (delay < 5ms) to ensure temporal correlation with the multimodal sensor data stream.

[0029] A timestamp alignment algorithm is used to synchronize the original multimodal sensing data stream with the surgical instrument motion trajectory data in time, eliminating the time delay between acquisition devices and generating a time-synchronized multi-source data stream; This step ensures that data from different sources match in the time dimension by unifying the time base, providing a consistent temporal basis for spatial analysis. The specific implementation is as follows: The core of timestamp alignment is to resolve clock discrepancies between devices. Although multispectral cameras, depth sensors, and optical positioning systems all use system time as a reference, the actual acquisition time varies (typically 5-50ms) due to startup delays and differences in hardware response. For example, the timestamp of a multispectral image is t1=14:30:05.123, the timestamp of the depth point cloud at the same physical moment may be t2=14:30:05.140 (17ms delay), and the timestamp of the instrument trajectory may be t3=14:30:05.130 (7ms delay). An algorithm is needed to align the three to the same time point.

[0030] The timestamp alignment algorithm combines Dynamic Time Warping (DTW) with linear interpolation: First, timestamp sequences for each data stream are established (e.g., T_img = [t1, t1+33ms, t1+66ms,...] for multispectral images, and T_track = [t3, t3+10ms, t3+20ms,...] for instrument trajectories); then, using the highest frequency data stream (instrument trajectory, 100fps) as the reference time axis, interpolation is performed on the other data streams—for multispectral images (30fps), if the reference time t is between t1 and t1+33ms, a virtual image frame at time t is generated through bilinear interpolation (preserving spectral features); for depth point clouds (20fps), virtual point clouds are also generated through interpolation between adjacent frames, ensuring that each reference time point has corresponding multispectral, depth, and trajectory data.

[0031] Algorithm parameter settings: The time window size is 50ms (i.e., only data within 50ms before and after the current time point is used for interpolation to avoid error accumulation from distant frames), and the interpolation error threshold is <5ms (if the time difference after interpolation exceeds 5ms, the frame is discarded and marked as an anomaly, and subsequently processed by the completion algorithm). For example, with a base time t=14:30:05.135, the multispectral image's t1=14:30:05.123 (difference of 12ms), and t1+33=14:30:05.156 (difference of 21ms), the image at time t is generated through interpolation, ensuring that the time deviation is <5ms.

[0032] The generated time-synchronized multi-source data stream contains corresponding multispectral images, depth point clouds, instrument positions, and orientations at each time point (10ms interval). For example, at t=14:30:05.130, the data combination is "multispectral image (interpolated) + depth point cloud (interpolated) + instrument coordinates (50.5, 30.6, 15.8) + orientation angles (0.2°, 2.1°, 1.6°)". After synchronization, time consistency needs to be verified by calculating the time difference of the same action (such as instrument contact with tissue) in different data sets, ensuring that all differences are <10ms to meet the real-time requirements of surgery.

[0033] Spatial coordinate unification processing is performed on time-synchronized multi-source data streams. Through coordinate transformation, all data are mapped to the same world coordinate system, ultimately generating a multi-dimensional perception data set of the surgical scene.

[0034] This step achieves spatial fusion of multimodal data by unifying the spatial benchmark and eliminating coordinate differences between different devices. The specific implementation is as follows: The prerequisite for unifying spatial coordinates is determining the transformation relationship between the coordinate systems of each device and the world coordinate system. The world coordinate system is defined as follows: the origin is located 1 meter above the center of the operating table, the x-axis is along the major axis of the operating table (head-to-foot direction), the y-axis is along the minor axis of the operating table (left-right direction), and the z-axis is perpendicular to the operating table and upwards (vertical direction). Multispectral cameras, depth sensors, and optical positioning systems each have independent local coordinate systems (e.g., the camera coordinate system has its origin at the lens optical center, and the z-axis is along the optical axis). These require calibration to obtain rigid transformation parameters from the local coordinate system to the world coordinate system—the rotation matrix R (a 3×3 matrix describing direction transformation) and the translation vector T (a 3×1 vector describing position offset).

[0035] The calibration process uses a high-precision calibration board (30×30cm in size, 1cm checkerboard spacing, positioning accuracy ±0.01mm): the calibration board is placed in multiple positions within the surgical area (at least 6 different orientations), and the coordinates of the calibration board in the local coordinate system of each device are recorded (e.g., the pixel coordinates of the calibration board corner points captured by the multispectral camera are converted into three-dimensional coordinates in the camera coordinate system; the depth sensor directly obtains the three-dimensional coordinates of the calibration board corner points; the optical positioning system tracks the marker points on the calibration board to obtain coordinates); the transformation parameters are solved by the least squares method to minimize the coordinate error of the calibration board in the world coordinate system (root mean square error <0.2mm). For example, the rotation matrix R_img of a multispectral camera is [[0.999, 0.01, 0.02], [-0.01, 0.999, -0.03], [-0.02, 0.03, 0.999]], and the translation vector T_img is [500, 300, 1000] mm, representing the position and orientation of the camera's optical center in the world coordinate system.

[0036] Coordinate transformation is achieved using the formula p_world = R × p_local + T, where p_local is the coordinate of the data in the local coordinate system, and p_world is the transformed world coordinate system coordinate. For example, the coordinates of a point in the depth point cloud in the sensor's local coordinate system are (10, 20, 30) mm. Substituting the transformation parameters R_depth and T_depth, we get the world coordinate system coordinates as (512, 321, 1035) mm. The position of the instrument in the optical tracking local coordinate system is (5, 3, 2) mm, which is transformed to (505, 303, 1002) mm.

[0037] After spatial unification, consistency verification is required: Select fixed points within the surgical area (such as reference points marked on the patient's skin), and obtain their world coordinates using multispectral images (derived from camera parameters), depth sensors, and optical positioning systems, respectively. Calculate the average deviation among the three, which must be <0.5mm. The final generated multidimensional sensing data set contains, at each time point (10ms interval), a multispectral image in the world coordinate system (pixel coordinates corresponding to world coordinates), a depth point cloud (x, y, z are all world coordinates), and instrument position and orientation (coordinates and angles in the world coordinate system), fully reflecting the spatiotemporal characteristics of the surgical scene and providing a unified spatial reference for subsequent 3D modeling.

[0038] S202, perform feature extraction and spatial registration on the multidimensional perception data set, construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structure, and decompose the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer. Specifically, tissue features from multispectral images and geometric features from depth point clouds can be extracted from multidimensional sensory datasets. Feature descriptor algorithms can be used to identify the boundaries of key anatomical structures and generate an initial feature set of anatomical structures. This step extracts core features that characterize anatomical structures by analyzing the optical and spatial information in multidimensional sensory data, laying the foundation for subsequent modeling. The specific implementation is as follows: Tissue feature extraction from multispectral images relies on the specific responses of different wavelengths to biological tissues. Multispectral images contain six wavelengths (450nm blue, 540nm green, 630nm red, 700nm near-infrared, 850nm near-infrared, and 940nm near-infrared). Among them, the 450nm wavelength has strong absorption of hemoglobin and can be used to distinguish blood vessels (the pixel value of vascular areas in this wavelength is 25%-35% lower than that of surrounding tissues); the 630nm wavelength is sensitive to bleeding areas (the pixel value of bleeding sites is 15%-20% higher than that of normal tissues due to low oxygen content); the 940nm wavelength can penetrate 1-2mm of superficial tissue and can identify the junction between fascia and muscle (the reflectivity of fascia in this wavelength is 10%-15% higher than that of muscle). During extraction, two-dimensional masks of tissues such as blood vessels, bleeding points, and fascia are generated by calculating the ratio features between bands (e.g., "vascular index = 450nm pixel value / 630nm pixel value", when the index < 0.65, it is determined to be a vascular region). Each mask marks the pixel range of the corresponding tissue in the image.

[0039] Geometric feature extraction from depth point clouds focuses on spatial morphological description, including surface normals (reflecting the orientation of a local surface, e.g., the rate of change of normal direction on a tumor surface >20° / mm, while that of normal tissue is <5° / mm), curvature (reflecting the degree of surface bending, with the radius of curvature of bone surfaces typically <8mm and that of adipose tissue >40mm), and point density (reflecting surface roughness, with the point density at the edge of a surgical incision reaching 600 points / cm). 2 Normal tissue has approximately 300 spots / cm 2 During the calculation, a neighborhood search with a radius of 3 mm is used (to ensure sufficient coverage of neighboring points to guarantee accuracy). The surface normal is solved by fitting the local plane using the least squares method, and the curvature is calculated by fitting the sphere (curvature value = 1 / radius of curvature). Finally, five-dimensional features (x, y, z, normal vector, curvature value) of each point cloud point are generated.

[0040] The feature descriptor algorithm uses the ORB (Oriented FAST and Rotated BRIEF) algorithm to identify key anatomical structure boundaries (such as the junction of the hepatic hilum and gallbladder, and vascular branching points). This algorithm first extracts candidate feature points through FAST corner detection (setting a threshold of 15; when the grayscale difference between a pixel and more than 12 of its 16 surrounding pixels is greater than 15, it is determined to be a corner point); then, it generates a 32-bit binary descriptor using the BRIEF algorithm (recording the grayscale comparison results of 256 pairs of pixels in the neighborhood of the feature point); finally, it filters stable boundary feature points using Hamming distance (distance < 40 is considered a match). For example, in liver surgery, the ORB algorithm can match the bleeding boundary in the 630nm band of a multispectral image with the curvature abrupt change in the depth point cloud, determining the boundary line between the liver and gallbladder within the surgical field of view, and generating an initial anatomical feature set (containing approximately 800-1200 feature points, each point associated with two-dimensional image coordinates and three-dimensional point cloud coordinates) including vascular pathways, organ contours, and key anatomical markers.

[0041] Based on the initial feature set of anatomical structures, the real-time acquired anatomical features are spatially registered with the pre-constructed standard anatomical model through the iterative nearest point algorithm to generate a registered accurate anatomical structure model. This step eliminates individual differences and imaging biases by aligning real-time anatomical features with standard models, thus establishing a precise anatomical benchmark. The specific implementation is as follows: The pre-built standard anatomical model is generated based on preoperative CT or MRI data. The Marching Cubes algorithm is used to convert the voxel data of the medical images (resolution 0.5mm×0.5mm×0.5mm) into a three-dimensional mesh model, which includes key structures such as the liver, gallbladder, and portal vein. The mesh contains approximately 150,000 mesh patches (to ensure high precision). The anatomical names of each structure (such as "left branch of portal vein" and "right hepatic vein") and spatial coordinate ranges (such as the starting coordinates of the main portal vein as (120, 80, 50) mm and the ending coordinates as (150, 90, 60) mm) are labeled and stored in STL format.

[0042] The Iterative Closest Point (ICP) algorithm is used to achieve spatial alignment between real-time features and the standard model. The process is as follows: First, select 1000 uniformly distributed feature points from the initial anatomical feature set as the target point set (avoiding point concentration in local areas). In the standard anatomical model, use a kd-tree nearest neighbor search (search radius 8mm) to find the nearest point for each target point, forming point pairs. Use Singular Value Decomposition (SVD) to calculate the rigid transformation matrix (containing rotation matrix R and translation vector T) that minimizes the root mean square error (RMSE) between point pairs, where R is a 3×3 matrix (describing orientation adjustment) and T is a 3×1 vector (describing positional offset). After updating the target point set using the transformation matrix, repeat the above process until the RMSE < 0.3mm or the number of iterations reaches 30 (balancing accuracy and efficiency). For example, the initial RMSE between real-time feature points on a patient's liver surface and the standard model was 2.8mm, which decreased to 0.25mm after 12 iterations, meeting the registration accuracy requirements.

[0043] The registered, precise anatomical model integrates the deep structural information of the standard model with real-time acquired surface features: preserving the three-dimensional course of the portal vein branches in the standard model (a key structure known preoperatively), overlaying liver surface depressions captured by real-time depth point clouds (new features formed intraoperatively), and correcting the model's material properties using tissue features from multispectral images (e.g., hemorrhage areas are marked in red, and normal liver tissue is marked in light red). The spatial coordinate error of the model needs to be calibrated using intraoperative verification points (e.g., preoperatively marked titanium clips) to ensure that the deviation between the position of the verification points in the model and their actual positions is <0.5mm, providing a reliable anatomical benchmark for subsequent dynamic modeling.

[0044] By combining surgical instrument motion trajectory data, the relative positional relationship between instruments and tissues is dynamically updated in the registered precise anatomical structure model, and a dynamic three-dimensional augmented reality model of key surgical anatomical structures is constructed. This step, by linking the spatial state of instruments and tissues in real time, enables the model to dynamically reflect the surgical procedure, as specifically implemented below: The surgical instrument motion trajectory data includes three-dimensional coordinates (x, y, z, in the world coordinate system, accuracy ±0.1 mm) and attitude angles (roll, pitch, yaw, accuracy ±0.3°) updated every 8 ms, which need to be spatially correlated with the registered anatomical model. First, the real-time position of the instrument tip (determined by the offset of the tip coordinates in the instrument's local coordinate system using the instrument CAD model, e.g., the offset of the scalpel tip is (0, 0, -50) mm) in the world coordinate system is calculated. Then, the shortest distance between the tip and the anatomical structure surface is calculated using the Euclidean distance formula (search range 30 mm, considering only surgically relevant structures): d = √[(x1-x2)]. 2 +(y1-y2) 2 +(z1-z2) 2 The coordinates are defined as follows: (x1, y1, z1) represent the coordinates of the instrument tip, and (x2, y2, z2) represent the coordinates of points on the anatomical structure surface. The interaction state is determined based on distance: d < 0.5 mm indicates "contact" (the instrument has touched the tissue), 0.5 mm ≤ d < 3 mm indicates "approaching" (about to contact), and d ≥ 3 mm indicates "far from" (no interaction). For example, the coordinates of the hemostat tip are (125.3, 82.1, 51.6) mm, and the coordinates of the nearest point on the liver surface are (125.5, 82.3, 51.8) mm. The calculated d = 0.3 mm, thus indicating "contact".

[0045] The construction of dynamic 3D augmented reality models requires real-time updates and clear visualization: the model's update frequency is synchronized with instrument trajectory data (updated every 8ms), ensuring no significant delay between the instrument position observed by the doctor and the actual operation (delay <10ms); hierarchical rendering technology is used, with anatomical structures displayed semi-transparently (50% transparency), and instruments displayed in solid color (red for contact, yellow for proximity, and green for distance), with distance values ​​marked at the closest point between the instrument and the tissue (e.g., "0.3mm"); local magnification is supported (up to 10x magnification), facilitating observation of the relative positions of fine structures (e.g., small blood vessels <2mm in diameter) and instruments. For example, in liver resection surgery, the model can display the relative position of the scalpel and the portal vein in real time, automatically magnifying the area when the distance is <2mm to remind the doctor to avoid damaging the blood vessel, thus constructing an augmented reality model that accurately reflects the dynamics of the surgery.

[0046] The dynamic 3D augmented reality model is decomposed hierarchically, separating the static anatomical structure feature layer and the dynamic instrument interaction feature layer to generate a dual-feature layer model representation.

[0047] This step, through layered processing, enables the model to both stably present the anatomical basis and highlight dynamic operational information. The specific implementation is as follows: The static anatomical feature layer contains core structures whose morphology remains largely unchanged during surgery, such as bones (ribs, spine), major blood vessels (aorta, inferior vena cava), and the fixed outlines of major organs (liver, kidney). This layer is based on a registered, accurate anatomical model and is updated only when permanent anatomical changes occur (e.g., after partial liver resection, the corresponding area's mesh is deleted from the model), with an update frequency of ≤1 time / minute (reducing computational resource consumption). For example, in kidney surgery, the static layer contains the spine, inferior vena cava, and kidney outline. Even if the kidney position shifts slightly during surgery, the kidney outline in the static layer is only updated after the shift is confirmed to be stable (e.g., positional change <1mm within 5 minutes), ensuring the stability of the anatomical baseline.

[0048] The dynamic instrument interaction feature layer contains elements that change in real time during surgery. In addition to the position and orientation of surgical instruments, it also includes real-time tissue deformation (such as indentation on the liver surface caused by instrument pressure, which is calculated by comparing the deformation amount in two consecutive depth points; updates are made when the deformation amount is >0.3mm) and bleeding area expansion (identified by changes in pixel values ​​in the 630nm band of multispectral images; for every 0.5cm increase in bleeding area). 2 The dynamic layer includes real-time updates and temporary markers (such as resection boundaries marked by doctors, indicated by dashed boxes, which can be adjusted via voice commands). This layer's update frequency is synchronized with the multidimensional sensing data (updated every 8ms), ensuring that dynamic information perfectly matches actual operations. For example, during electrocoagulation hemostasis, the dynamic layer displays the shrinking process of the bleeding area in real time and marks the distance between the electrocoagulation device and the bleeding point (e.g., "1.2mm"), helping doctors operate precisely.

[0049] The dual-feature-layer model is implemented through a layered rendering pipeline: a static layer is rendered first (as the background), and a dynamic layer is overlaid on top of the static layer (highlighting the dynamic content). The two layers are strictly aligned using a world coordinate system (ensuring no spatial deviation between instrument positions and anatomical structures). Surgeons can switch display modes via an interactive interface: displaying the static layer separately for surgical path planning, displaying the dynamic layer separately for instrument tracking, or displaying them in combination to balance the anatomical background and real-time operation. This layered design ensures both the stability of the anatomical structures and highlights the dynamism of the surgical process, providing a clear and comprehensive model foundation for subsequent navigation and evaluation.

[0050] S203, Based on digital twin technology, the anatomical structure feature layer and the instrument interaction feature layer are dynamically matched and rendered in real time to generate augmented reality navigation information and operation guidance scheme; Specifically, a digital twin of the surgical scene can be created by importing the anatomical structure feature layer and the instrument interaction feature layer into the digital twin engine to generate an initial digital twin model; This step provides a digital foundation for subsequent dynamic matching and navigation by constructing a virtual mapping of the physical surgical scene. The specific implementation is as follows: A digital twin of a surgical scene is a precise virtual reproduction of the physical surgical environment, requiring digital representations of elements such as patient anatomy, surgical instruments, and the operating room environment (e.g., the position of surgical lights, the boundaries of sterile areas). During construction, static environmental data of the operating room (e.g., operating table dimensions of 1.8m × 0.6m, ceiling height of 3m) is first collected, and an environmental mesh model (accuracy ±5mm) is generated through 3D scanning. Then, the patient's preoperative imaging data (CT / MRI) and real-time intraoperative perception data are integrated to ensure that the twin includes both deep anatomical structures (e.g., the 3D course of intracranial blood vessels) and reflects dynamic changes during surgery (e.g., real-time displacement of brain tissue).

[0051] The import of the anatomical structure feature layer and the instrument interaction feature layer requires a unified data format and coordinate system. The anatomical structure feature layer (static layer) is stored in STL format and contains a mesh model of structures such as the liver and gallbladder (each structure is accompanied by an anatomical label, such as "right lobe of liver" and "left branch of portal vein"). During import, coordinate transformation (aligned with the world coordinate system, error <0.3mm) is used to ensure consistency with the physical location. The instrument interaction feature layer (dynamic layer) is transmitted in real time in JSON format and contains the instrument ID (such as "surgical knife-01"), the three-dimensional coordinates of the tip (x=120.5mm, y=85.3mm, z=50.2mm), attitude angles (roll angle 3°, pitch angle 5°, yaw angle 2°), and interaction status (such as "contact with liver surface"). The import frequency is synchronized with the intraoperative data (updated every 10ms).

[0052] The digital twin engine utilizes a customized medical-grade engine based on Unreal Engine, supporting real-time rendering of millions of polygons (frame rate ≥ 60fps). It features a data fusion interface (compatible with multispectral images, depth point clouds, and trajectory data) and a physics engine (simulating collision feedback between instruments and tissues). After the initial digital twin model is generated, it needs to be validated using benchmarks (such as the three titanium clips placed during surgery, with the deviation between the marker positions in the virtual model and their actual positions < 0.5mm) to ensure consistency between the virtual and physical scenes. For example, in laparoscopic cholecystectomy, the initial model must clearly show the spatial relationship between the gallbladder and the common hepatic duct, as well as the real-time position of the laparoscopic instruments, providing a reliable starting point for subsequent dynamic matching.

[0053] In the initial digital twin model, dynamic matching of the anatomical structure feature layer and the instrument interaction feature layer is realized. The interaction state between the instrument and the anatomical structure is calculated in real time through a collision detection algorithm to generate dynamic matching results. This step provides interactive data for navigation by tracking the spatial relationship between instruments and anatomical structures in real time, as specifically implemented as follows: The core of dynamic matching is maintaining the spatial correlation between the instrument interaction feature layer and the anatomical structure feature layer. Specifically, when the anatomical structure undergoes a slight displacement due to respiration or manipulation (e.g., the liver moves ±3mm with respiration), the instrument feature layer must synchronously update its relative position. In implementation, this is achieved by comparing key points on the anatomical structure surface (e.g., the bifurcation points of blood vessels on the liver surface) in two consecutive frames of depth point cloud data in real time. The displacement vector (Δx, Δy, Δz) of the structure is calculated, and this vector is then applied to the instrument interaction feature layer to ensure that the relative positional error between the instrument and the tissue is <0.2mm. For example, when the liver moves upward by 2mm due to the patient's inhalation, the coordinates of the instrument tip relative to the liver are automatically corrected to maintain the interaction state of "1mm distance from the liver surface."

[0054] The collision detection algorithm employs the GJK (Gilbert-Johnson-Keerthi) algorithm. This algorithm detects the interaction state by calculating the Minkowski difference (a geometric operation used to determine whether two convex bodies intersect) between the instrument and the anatomical structure. It does not require traversing all mesh surfaces, resulting in high computational efficiency (single detection time <1ms), making it suitable for real-time scenarios. Algorithm parameters include: detection frequency (synchronized with instrument trajectory updates, once every 10ms), safety distance threshold (set according to structural importance, such as 2mm for portal vein and 5mm for ordinary liver tissue), and collision depth threshold (>0.1mm is considered "collision").

[0055] The interaction status is divided into three categories: "Far Away" (distance between instrument and structure > safety threshold, such as a scalpel 5mm from the gallbladder, with a safety threshold of 3mm, judged as far away), "Close Away" (distance ≤ safety threshold and no collision, such as a distance of 1.5mm, judged as close, requiring a warning), and "Collision" (geometric intersection or distance < 0, such as an instrument penetrating 0.3mm into liver tissue, judged as a collision). The dynamic matching results are output as structured data, including timestamp, instrument ID, target structure name, distance value (accurate to 0.01mm), and interaction status label, such as "2025-10-24 15:20:30.123, scalpel-01, left branch of portal vein, distance 1.2mm, status: close," providing quantitative basis for subsequent rendering and guidance.

[0056] Based on the dynamic matching results, a real-time rendering pipeline is used to visualize the digital twin model, overlaying key anatomical structures and instrument path guidance to generate an augmented reality navigation view. This step uses visualization technology to transform abstract data into intuitive images, helping doctors quickly understand the surgical scenario. The specific implementation is as follows: The real-time rendering pipeline comprises three stages: geometry processing, lighting calculation, and shading output. The geometry processing stage simplifies the mesh model of the anatomical structure feature layers (preserving key details, such as branches with blood vessel diameters >1mm; the number of triangles is reduced by 30% after simplification to ensure rendering efficiency). The lighting calculation stage employs a global illumination algorithm (simulating the scattering effect of surgical lights, with light intensity decreasing with distance; for example, the intensity is 1000 lux at 1m from the light source and 250 lux at 2m) to enhance the model's three-dimensionality. The shading output stage assigns specific materials to different structures (e.g., the liver is a light red semi-transparent material with 50% transparency; blood vessels are a red opaque material with 30% reflectivity) to differentiate tissue types.

[0057] Key anatomical structures are highlighted using a multi-level visual enhancement strategy: For high-risk structures (such as the portal vein and bile ducts), a 3mm wide yellow outline is superimposed on the model surface, with the brightness of the outline dynamically changing with the distance from the instruments (brightness increases by 50% when the distance is less than the safety threshold); for target operating areas (such as the tumor resection boundary), a blue semi-transparent mask is used to cover them (30% transparency), with the mask edge flashing as a cue (frequency 2Hz); for areas where operations have been completed (such as wounds after hemostasis), they are marked in green, and the transparency is reduced to 20% to minimize visual interference. For example, in liver tumor resection, the tumor boundary is highlighted with a blue mask, and blood vessels within a 5mm diameter are marked with a yellow outline; when the scalpel approaches the blood vessel to within 1.5mm, the brightness of the outline doubles.

[0058] The instrument path guidance includes a planned path and real-time deviation prompts: the planned path is generated based on preoperative planning and intraoperative dynamic adjustments, represented by a green dashed line (2mm wide), with a directional arrow marked every 10mm to indicate the direction of movement; when the deviation between the actual instrument trajectory and the planned path is >1mm, the deviation segment automatically turns yellow; when the deviation is >2mm, it turns red, and a numerical label is displayed at the point of maximum deviation (e.g., "deviation 2.3mm"). In addition, a virtual "target point" (a small red ball, 1mm in diameter) is generated 5mm in front of the instrument tip to guide the surgeon along the planned path. For example, during laparoscopic suturing, the target point indicates the insertion position of the next stitch in advance, and the path dashed line updates in real time as the instrument moves.

[0059] Augmented reality navigation views are displayed via head-mounted displays (HUDs) or surgical microscope eyepieces. The views must meet the following requirements: superposition error between the virtual model and the real scene <1mm (ensuring this through optical calibration), latency <20ms (to avoid operational lag), and a field of view ≥80° (covering the surgical field of view). For example, in neurosurgery, the navigation view can overlay a 3D model of intracranial blood vessels onto the surgeon's field of vision, highlighting vessels within 3mm of the surgical instruments, and guiding the instruments along a safe path to the tumor location in real time.

[0060] Based on the augmented reality navigation view and surgical procedure knowledge base, an operation guidance plan containing operation steps, risk warnings and alternative solutions is automatically generated, and finally, a complete augmented reality navigation information and operation guidance plan is output.

[0061] This step integrates navigation information with clinical guidelines to provide intelligent guidance for surgical procedures, specifically as follows: The Surgical Procedure Knowledge Base is a structured collection of clinical guidelines, categorized by surgical type (e.g., "Laparoscopic Cholecystectomy" and "Glioma Resection"). Each entry includes operative steps (e.g., "Dissection of the Calot's Triangle" and "Clamping of the Cystic Duct"), key parameters (e.g., "Clamping force must reach 30N, lasting 5 seconds"), contraindications (e.g., "Conventional clamps are prohibited when the diameter of the cystic duct is >5mm"), and complication management (e.g., "Immediate electrocoagulation for hemostasis upon bleeding, power set to 30W"). The knowledge base is constructed using an ontology (class: surgical step; attributes: sequence number, name, time, preconditions) and supports semantic retrieval (e.g., inputting "Calot's Triangle bleeding" returns the corresponding treatment steps).

[0062] Operation steps are generated based on a match between the current surgical stage and the navigation view: by analyzing instrument interaction status (e.g., "instrument located in the Calot's triangle") and completed steps (e.g., "cystic artery exposed"), the next stage step is matched from the knowledge base, for example, "Step 3: bluntly dissect the connective tissue between the cystic duct and the common hepatic duct with dissecting forceps, limiting the operation area to a 5mm region within the Calot's triangle," and the expected time is indicated (e.g., "suggested completion time: 5min"). The step description needs to incorporate spatial information from the augmented reality navigation view to clearly define the operation location (e.g., "perform dissection within the blue highlighted area in the navigation view").

[0063] Risk warnings are triggered based on dynamic matching results and are divided into three levels: Level 1 warnings (low risk, such as "device deviates from planned path by 1.2mm, fine-tuning recommended"), displayed in yellow text in the corner of the view; Level 2 warnings (medium risk, such as "1.5mm from left branch of portal vein, approaching safety threshold"), with flashing text and slight vibration feedback (headset vibration intensity 20%); Level 3 warnings (high risk, such as "contact with hepatic artery, potentially causing bleeding"), with large red text covering the center of the view, accompanied by continuous vibration (intensity 50%) and paused path guidance until the risk is eliminated. Each warning includes an explanation of the cause (e.g., "Cause of risk: distance between device tip and artery surface < 0.5mm") and avoidance suggestions (e.g., "Recommend moving 0.8mm along the negative y-axis").

[0064] Alternative solutions are generated when the original solution is not feasible (e.g., "the original path is blocked by a bleeding area"). These alternative paths are simulated in a digital twin model using a Monte Carlo tree search algorithm (search depth 5 steps, considering 8 possible directions per step). The safety (probability of avoiding high-risk structures > 90%) and feasibility (instrument accessibility > 80%) of the paths are evaluated, and 2-3 optimal solutions are selected. For example, when the original path for tumor resection requires crossing a large blood vessel, an alternative solution might suggest "circling counterclockwise along the tumor edge, increasing the operation time by approximately 3 minutes, but reducing the risk of blood vessel damage by 70%," and this alternative path would be displayed as an orange dashed line in the navigation view.

[0065] A complete augmented reality navigation information and operation guidance solution is an integration of navigation view and guidance text, synchronously output through a head-mounted display: the left side of the view shows the current step and remaining time, the right side shows risk warnings and alternative solutions, and the center displays an augmented reality overlaid surgical scene. The solution supports voice interaction (e.g., when a doctor says "Show alternative solution 2," the view automatically switches to the corresponding path) and records the operation process in real time (saving navigation data every 30 seconds) to provide a basis for postoperative evaluation. For example, the final output in neurosurgery is as follows: the central view displays an augmented reality model of the brain tumor and surrounding blood vessels, the left side prompts "Step 4: Remove the tumor core along the green path, 8 minutes remaining," the right side warns "2mm from the motor cortex, be careful to protect the functional area," and includes a brief description of an alternative path.

[0066] S204, perform multi-dimensional quantitative analysis of the surgical operation process based on the augmented reality navigation information and operation guidance scheme, and output a quantitative evaluation result including operation accuracy and execution efficiency.

[0067] Specifically, it can record the actual movement trajectory of instruments during the surgical procedure in real time, and compare and analyze it with the expected path in the augmented reality navigation information to generate instrument movement deviation data; This step provides raw data for subsequent accuracy assessment by tracking the difference between the actual movement of the device and the expected path. The specific implementation is as follows: The real-time recording of the actual movement trajectory of the instrument relies on the intraoperative positioning system. This system uses infrared markers (3 reflective points arranged in a triangle, spaced 2cm apart) installed on the instrument. A binocular infrared camera (frame rate 100Hz, positioning accuracy ±0.1mm) collects the three-dimensional coordinates (x, y, z, based on the world coordinate system, with the origin at the center of the operating table) every 10ms. At the same time, it records the attitude angles (roll angle, pitch angle, yaw angle, accuracy ±0.3°). For example, the coordinates of a scalpel at time t1=10:20:05.123 are (150.2, 80.5, 45.3) mm, and the attitude angle is (2.1°, 3.5°, 1.2°); the coordinates at time t2=10:20:05.133 are (150.5, 80.7, 45.4) mm, and the attitude angle is (2.2°, 3.6°, 1.3°). Continuous recording forms a trajectory sequence, including the changes in position and attitude over time.

[0068] The expected path in augmented reality navigation information originates from preoperative planning and intraoperative dynamic adjustments: Preoperatively, a core path (such as a safe passage for tumor resection) is planned based on CT / MRI data, represented as a three-dimensional polyline. Each polyline segment includes the coordinates of the starting and ending points and a suggested movement speed (e.g., 5 mm / s). Intraoperatively, the path is dynamically corrected based on real-time displacement of anatomical structures (e.g., the liver moves ±3 mm with respiration) through a digital twin model, with the correction frequency synchronized with anatomical structure updates (once every 50 ms). For example, the initial expected path for liver tumor resection is a straight line from (140.0, 75.0, 40.0) mm to (160.0, 85.0, 50.0) mm. Intraoperatively, due to the liver shifting upwards by 2 mm, the path is corrected to (140.0, 75.0, 42.0) mm to (160.0, 85.0, 52.0) mm.

[0069] Comparative analysis requires dual alignment in both time and space: Temporally, timestamp matching (allowing ±5ms error) is used to associate the actual trajectory with the expected path at the same point in time. For example, the point on the actual trajectory at t=10:20:05.123 corresponds to the point on the expected path at t=10:20:05.125. Spatially, the Euclidean distance between corresponding points is calculated (formula: d=√[(x_actual - x_expected)]). 2 +(y actual - y expected) 2 +(z actual - z pre-) 2The instantaneous deviation is obtained, and the direction of the deviation is recorded (e.g., a deviation of 0.3 mm in the positive x-axis direction). For example, the expected coordinates of the scalpel at time t1 are (150.0, 80.5, 45.5) mm, and the deviation d is calculated as √[(0.2)]. 2 +(0) 2 +(-0.2) 2 ]≈0.28mm, with the direction being the positive x-axis and the negative z-axis.

[0070] The generated instrument motion deviation data includes a timestamp, instrument ID, actual coordinates, expected coordinates, instantaneous deviation value, and deviation direction, for example, "10:20:05.123, scalpel-01, (150.2, 80.5, 45.3), (150.0, 80.5, 45.5), 0.28mm, x+0.2, z-0.2". The data needs to be stored in segments according to surgical steps (e.g., "separation stage" and "resection stage"), with each segment containing at least 500 data points (to ensure statistical reliability), providing continuous and structured input for subsequent accuracy index calculations.

[0071] Based on the instrument motion deviation data, the instrument positioning accuracy, path tracking error and operational stability index are calculated to generate an operational accuracy evaluation dataset. This step evaluates operational accuracy from multiple dimensions by quantitatively analyzing deviation data, and is implemented as follows: Instrument positioning accuracy reflects the precision with which the instrument tip reaches the target location. The calculation focuses on the final positioning deviation of each key target point (such as tumor boundary points or vascular branch points). First, records of instrument arrival at the target point are selected from the deviation data (judgment criteria: instrument speed < 0.5 mm / s and dwell time > 500 ms). Then, the Euclidean distance between the actual coordinates of that point and the target coordinates is calculated to obtain the single-point positioning deviation. Next, the mean deviation (μ = Σd_i / n, where n is the number of target points) and the maximum deviation (d_max = max(d_i)) of all target points are statistically analyzed. For example, in brain tumor resection, if the positioning deviations of three target boundary points are 0.3 mm, 0.5 mm, and 0.4 mm, the mean μ = 0.4 mm, and the maximum deviation d_max = 0.5 mm. Typically, a mean deviation < 1 mm and a maximum deviation < 2 mm are required for successful positioning.

[0072] The path tracking error assessment evaluates the overall degree to which the device deviates from the expected path during movement, using two indicators: root mean square error (RMSE) and maximum tracking error. The RMSE calculation formula is √(Σd_i). 2 / n), where d_i is the instantaneous deviation of each sampling point on the path, and n is the total number of sampling points on the path; the maximum tracking error is the maximum instantaneous deviation throughout the entire path. For example, a planned path of 50mm in length contains 500 sampling points, and the total sum of squared deviations is 50mm. 2 Therefore, RMSE = √(50 / 500) = √0.1 ≈ 0.32 mm; the maximum instantaneous deviation throughout the entire process is 0.8 mm, indicating that the path tracking is generally stable. In addition, it is also necessary to calculate the proportion of deviations exceeding the threshold (such as the proportion of points with a deviation > 1 mm to the total number of points, which should be < 5%), to reflect the consistency of path tracking.

[0073] Operational stability measures the smoothness of instrument movement and is calculated using the coefficient of variation (CV = standard deviation / mean) and acceleration fluctuation values ​​of the deviation data. The coefficient of variation reflects the dispersion of the deviation; CV < 0.3 indicates high stability (small deviation fluctuation). Acceleration fluctuation values ​​are calculated by taking the acceleration change over three consecutive sampling points (a = Δv / Δt, Δt = 10 ms), and statistical fluctuation values ​​> 5 mm / s are considered. 2 The number of times the fluctuation exceeds the threshold is considered stable if the percentage of such occurrences is less than 10%. For example, if the mean deviation of a certain operation is 0.4 mm and the standard deviation is 0.1 mm, then the CV = 0.25 (high stability); the number of times the acceleration fluctuation exceeds the threshold is 20, and the total number of sampling points is 200, which accounts for 10%, just meeting the stability requirement.

[0074] The operational accuracy assessment dataset integrates the above indicators and is categorized by surgical steps (e.g., "dissection of the gallbladder triangle" and "vascular clamping"). Each step includes positioning accuracy (mean, maximum deviation), path tracking error (RMSE, maximum error, percentage exceeding threshold), and operational stability (CV, percentage of acceleration fluctuations). For example, "Dissection stage: positioning mean 0.4mm, maximum 0.6mm; RMSE 0.3mm, maximum 0.8mm, percentage exceeding threshold 3%; CV 0.2, percentage of acceleration fluctuations 8%." The dataset should include the normal ranges for the indicators (e.g., positioning mean <0.8mm is considered excellent) to provide a reference for subsequent comprehensive evaluation.

[0075] Based on the time requirements of the operation guidance plan, analyze the difference between the actual time consumption and the standard time consumption of each operation step, calculate the time efficiency and process continuity indicators, and generate an execution efficiency evaluation dataset. This step, through time-based analysis, assesses the rhythm and rationality of the surgical procedure, specifically implemented as follows: The timeframes in the operational guidelines are defined by the surgical procedure knowledge base. Each step includes a standard time (Tstandard) and an allowable fluctuation range (typically ±20%). For example, the standard time for the "dissection of the Calot's Triangle" step is 5 minutes (300 seconds), with an allowable range of 240-360 seconds; the standard time for the "tumor resection" step is 10 minutes (600 seconds), with an allowable range of 480-720 seconds. The timeframes also include requirements for the connection between steps (such as "suture preparation must be performed within 30 seconds after hemostasis is completed") to ensure a smooth workflow.

[0076] The actual time consumption is recorded through instrument interaction status: when the instrument enters the corresponding operating area (such as the Calot's triangle) and begins to perform characteristic actions (such as the opening and closing action of the dissecting forceps > 3 times), the start time of the step (tstart) is recorded; when the operation guide determines that the step is completed (such as "complete exposure of the cystic duct"), the end time of the step (tend) is recorded. The actual time consumption Tactual = tend - tstart. For example, for the "dissection of the Calot's triangle" step, tstart = 10:25:00, tend = 10:30:30, and the actual time consumption is 330 seconds (5.5 minutes).

[0077] Time efficiency metrics include time deviation rate and efficiency score. Time deviation rate = (Tactual - Tstandard) / Tstandard × 100%, where a positive value indicates exceeding the time limit and a negative value indicates achieving the time limit ahead of schedule. The efficiency score is assigned according to the deviation rate range (e.g., deviation rate < 5% = 100 points, 5%-10% = 90 points, 10%-20% = 70 points, > 20% = 50 points). For example, in the above separation step, Tactual = 330 seconds, Tstandard = 300 seconds, deviation rate = 10%, and efficiency score = 90 points. For critical steps (such as tumor resection), the score weight is increased by 1.2 times to ensure priority for the efficiency of core operations.

[0078] The process coherence index assesses the smoothness of transitions between steps, including transition delay time and the number of interruptions. Transition delay time is the difference between the actual interval (tinterval) from the end of one step to the start of the next and the standard interval (Tinterval, e.g., 30 seconds). A delay is considered to be greater than 10 seconds if tinterval - Tinterval > 10 seconds. The number of interruptions is the number of times the instrument leaves the operating area during the execution of a step (each departure exceeding 5 seconds is considered an interruption). For example, from the end of the "hemostasis" step to the start of the "suture" step, tinterval = 45 seconds, Tinterval = 30 seconds, resulting in a delay of 15 seconds; during suture, the instrument leaves the operating area twice, resulting in 2 interruptions. The coherence score = 100 - (delay seconds × 1 + number of interruptions × 5). The score for the above example is 100 - (15 × 1 + 2 × 5) = 75 points.

[0079] The execution efficiency evaluation dataset integrates time efficiency and consistency metrics step-by-step. Each step includes actual time, standard time, deviation rate, efficiency score, connection delay, number of interruptions, and consistency score. For example, "Tumor resection: Tactual = 660 seconds, Tstandard = 600 seconds, deviation rate 10%, efficiency score 90; connection delay 5 seconds, 1 interruption, consistency score 90." The dataset needs to be labeled with the importance weight of each step (e.g., core steps weight 1.5, auxiliary steps 0.8) to provide a basis for the comprehensive score.

[0080] By integrating the operational accuracy assessment dataset and the execution efficiency assessment dataset, a comprehensive score is generated through a multi-index fusion algorithm, and the final output is a quantitative assessment result that includes both operational accuracy and execution efficiency.

[0081] This step integrates multi-dimensional indicators to form a comprehensive surgical procedure assessment, specifically implemented as follows: The multi-index fusion algorithm uses a weighted summation method. First, it standardizes each index of accuracy and efficiency (mapping it to 0-100 points), and then calculates the total score according to preset weights. The weight allocation is based on the Delphi method (the average score is taken by 5 senior surgeons), with operational accuracy accounting for 60% (positioning accuracy 20%, path tracking error 25%, operational stability 15%) and execution efficiency accounting for 40% (time efficiency 25%, process continuity 15%). For example, if the average positioning accuracy is 0.4mm (corresponding to 85 points), the path tracking RMSE is 0.3mm (corresponding to 90 points), and the operational stability CV is 0.2 (corresponding to 95 points), then the accuracy score = 85×20% + 90×25% + 95×15% = 17 + 22.5 + 14.25 = 53.75 points; the time efficiency score is 90 points, the consistency score is 90 points, the efficiency score = 90×25% + 90×15% = 22.5 + 13.5 = 36 points; the overall score = 53.75 + 36 = 89.75 points (rounded to 90 points).

[0082] The overall score needs to be weighted according to the type of surgery. For example, neurosurgery has higher precision requirements (precision weight 70%), while laparoscopic cholecystectomy is more sensitive to efficiency (efficiency weight 50%). Furthermore, a deduction mechanism is implemented for exceeding threshold indicators (such as maximum positioning deviation > 2mm), deducting 5 points for each occurrence to ensure key indicators meet requirements. For example, a surgery might have an overall score of 85, but if the path tracking exceeds the threshold by 8% (standard < 5%), the final score would be 85 - 5 = 80 points.

[0083] The quantitative evaluation results output includes three parts: detailed individual indicators (listing all raw indicators and scores for accuracy and efficiency, such as "mean positioning 0.4mm, score 85"), comprehensive score and grade (scores of 90-100 are "Excellent", 80-89 are "Good", 70-79 are "Pass", and <70 are "Needs Improvement"), and improvement suggestions (proposing optimization directions for indicators with low scores, such as "path tracking exceeding the threshold by 3%, it is recommended to reduce unnecessary fine-tuning of instruments"). The results are presented in a structured report format and synchronized to the surgical record system to support postoperative review by doctors (such as comparing score changes in different surgeries performed by the same doctor) and teaching and training (such as analyzing the differences in indicators between novice doctors and experts).

[0084] For example, the quantitative evaluation results of a liver resection surgery were as follows: Among the individual indicators, the average positioning accuracy was 0.5 mm (80 points), the path tracking RMSE was 0.4 mm (85 points), the operation stability CV was 0.25 (88 points), the time efficiency score was 85 points, and the consistency score was 90 points; the overall score was 86 points, and the grade was "good"; the improvement suggestion was "the maximum positioning deviation was 0.9 mm, close to the threshold of 1 mm, and it is recommended to add pause calibration at key target points." This result not only objectively reflects the operation level, but also provides specific directions for subsequent improvement, realizing the quantitative closed-loop management of surgical operations.

[0085] As can be seen, real-time acquisition of multispectral image data and depth point cloud data of the surgical area within the operating room generates a multidimensional perception data set of the surgical scene. Feature extraction and spatial registration are performed on the multidimensional perception data set to construct a dynamic 3D augmented reality model of the key surgical anatomical structures. This dynamic 3D augmented reality model is then decomposed into an anatomical structure feature layer and an instrument interaction feature layer. Based on digital twin technology, the anatomical structure feature layer and the instrument interaction feature layer are dynamically matched and rendered in real time to generate augmented reality navigation information and operation guidance schemes. Based on the augmented reality navigation information and operation guidance schemes, the surgical operation process is subjected to multidimensional quantitative analysis, and quantitative evaluation results are output. This provides augmented reality navigation guidance that is synchronized with the real surgical scene in real time, while also providing accurate quantitative evaluation of the surgical operation, thereby improving the accuracy, safety, and trainability of the surgery.

[0086] Another embodiment of the present invention provides an augmented reality intelligent navigation and operation quantification evaluation system in the operating room, see [link to relevant documentation]. Figure 3 The system may include: The acquisition module 301 is used to acquire multispectral image data and depth point cloud data of the surgical area in the operating room in real time, and combine them with surgical instrument motion trajectory data to generate a multidimensional perception data set of the surgical scene. Extraction module 302 is used to extract features and spatially register the multidimensional perception data set, construct a dynamic three-dimensional augmented reality model of the key surgical anatomical structure, and decompose the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer. The generation module 303 is used to dynamically match and render the anatomical structure feature layer and the instrument interaction feature layer in real time based on digital twin technology, and generate augmented reality navigation information and operation guidance scheme. The evaluation module 304 is used to perform multi-dimensional quantitative analysis of the surgical operation process based on the augmented reality navigation information and operation guidance scheme, and output quantitative evaluation results including operation accuracy and execution efficiency.

[0087] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0088] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0089] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.

[0090] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A method for augmented reality intelligent navigation and operation quantitative evaluation in an operating room, characterized in that, The method comprises: Real-time acquisition of multispectral image data and depth point cloud data of the surgical area in the operating room, and generation of a multi-dimensional perception data set of the surgical scene in combination with surgical instrument motion trajectory data; Feature extraction and spatial registration of the multi-dimensional perception data set, construction of a dynamic three-dimensional augmented reality model of the key anatomical structure of the surgery, and decomposition of the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer; Dynamic matching and real-time rendering of the anatomical structure feature layer and the instrument interaction feature layer based on digital twin technology, generation of augmented reality navigation information and operation guidance scheme; Multi-dimensional quantitative analysis of the operation process according to the augmented reality navigation information and operation guidance scheme, and output of quantitative evaluation results including operation accuracy and execution efficiency.

2. The method of claim 1, wherein, The real-time acquisition of multispectral image data and depth point cloud data of the surgical area in the operating room, and the generation of a multi-dimensional perception data set of the surgical scene in combination with surgical instrument motion trajectory data, comprises: Synchronously acquiring a multispectral image sequence and a depth point cloud sequence of the surgical area to generate an original multi-modal sensing data stream; Real-time capture of spatial position and attitude changes of the surgical instrument to generate surgical instrument motion trajectory data; Time stamp alignment algorithm is adopted to synchronize the original multi-modal sensing data stream and the surgical instrument motion trajectory data in time sequence, eliminate the time delay between the acquisition devices, and generate a time-synchronized multi-source data stream; Spatial coordinate unification processing is performed on the time-synchronized multi-source data stream, and all data are mapped to the same world coordinate system through coordinate transformation, and finally a multi-dimensional perception data set of the surgical scene is generated.

3. The method of claim 2, wherein, The feature extraction and spatial registration of the multi-dimensional perception data set, the construction of the dynamic three-dimensional augmented reality model of the key anatomical structure of the surgery, and the decomposition of the dynamic three-dimensional augmented reality model into the anatomical structure feature layer and the instrument interaction feature layer, comprise: Extracting tissue features of the multispectral image and geometric features of the depth point cloud from the multi-dimensional perception data set, and identifying the boundary of the key anatomical structure by using a feature descriptor algorithm to generate an initial anatomical feature set; Based on the initial anatomical feature set, the real-time collected anatomical features are spatially registered with the pre-constructed standard anatomical model by using an iterative closest point algorithm to generate an accurate anatomical structure model after registration; In combination with the surgical instrument motion trajectory data, the relative position relationship between the instrument and the tissue in the accurate anatomical structure model after registration is dynamically updated to construct a dynamic three-dimensional augmented reality model of the key anatomical structure of the surgery; The dynamic three-dimensional augmented reality model is hierarchically decomposed to separate the static anatomical structure feature layer and the dynamic instrument interaction feature layer to generate a double-feature layer model representation.

4. The method of claim 3, wherein, The dynamic matching and real-time rendering of the anatomical structure feature layer and the instrument interaction feature layer based on digital twin technology, and the generation of augmented reality navigation information and operation guidance scheme, comprise: Establishing a digital twin of the surgical scene, importing the anatomical structure feature layer and the instrument interaction feature layer into a digital twin engine to generate an initial digital twin model; The dynamic matching of the anatomical structure feature layer and the instrument interaction feature layer is realized in the initial digital twin model, the interaction state of the instrument and the anatomical structure is calculated in real time through a collision detection algorithm, and a dynamic matching result is generated; Based on the dynamic matching result, the digital twin model is visualized by using a real-time rendering pipeline, key anatomical structure highlights and instrument path guides are superimposed, and an augmented reality navigation view is generated; According to the augmented reality navigation view and the surgical procedure knowledge base, an operation guidance scheme including operation steps, risk prompts and alternative schemes is automatically generated, and finally complete augmented reality navigation information and operation guidance schemes are output.

5. The method of claim 4, wherein, The operation process is analyzed in multiple dimensions according to the augmented reality navigation information and the operation guidance scheme, and quantitative evaluation results including operation accuracy and execution efficiency are output, including: The actual motion trajectory of the instrument during the operation process is recorded in real time, and is compared and analyzed with the expected path in the augmented reality navigation information, and instrument motion deviation data is generated; Based on the instrument motion deviation data, instrument positioning accuracy, path tracking error and operation stability indicators are calculated, and operation accuracy evaluation data sets are generated; Combined with the time node requirements of the operation guidance scheme, the differences between the actual time consumption and the standard time consumption of each operation step are analyzed, the time efficiency and the process coherence indicators are calculated, and the execution efficiency evaluation data set is generated; The operation accuracy evaluation data set and the execution efficiency evaluation data set are integrated, a comprehensive score is generated through a multi-index fusion algorithm, and finally the quantitative evaluation results including operation accuracy and execution efficiency are output.

6. An augmented reality intelligent navigation and operation quantitative evaluation system in an operating room, characterized in that, The system comprises: The acquisition module is configured to acquire multi-spectral image data and depth point cloud data of a surgical area in an operating room in real time, and combine instrument motion trajectory data to generate a multi-dimensional perception data set of a surgical scene; The extraction module is configured to perform feature extraction and spatial registration on the multi-dimensional perception data set, construct a dynamic three-dimensional augmented reality model of key anatomical structures of the surgery, and decompose the dynamic three-dimensional augmented reality model into an anatomical structure feature layer and an instrument interaction feature layer; The generation module is configured to dynamically match the anatomical structure feature layer and the instrument interaction feature layer based on digital twin technology and perform real-time rendering to generate augmented reality navigation information and operation guidance schemes; The evaluation module is configured to analyze the operation process in multiple dimensions according to the augmented reality navigation information and the operation guidance scheme, and output quantitative evaluation results including operation accuracy and execution efficiency.

7. The system of claim 6, wherein, The acquisition module is specifically configured to: Synchronously acquire a multi-spectral image sequence and a depth point cloud sequence of the surgical area to generate original multi-modal sensing data streams; Capture spatial position and attitude changes of the surgical instrument in real time to generate instrument motion trajectory data; Use a timestamp alignment algorithm to time-synchronize the original multi-modal sensing data streams and the instrument motion trajectory data, eliminate time delays between acquisition devices, and generate time-synchronized multi-source data streams; Uniformly process the time-synchronized multi-source data streams in spatial coordinates, map all data to the same world coordinate system through coordinate transformation, and finally generate a multi-dimensional perception data set of the surgical scene.

8. The system of claim 7, wherein, The extraction module is specifically configured to: The multi-spectral image and the depth point cloud are extracted from the multi-dimensional perception data set, a feature descriptor algorithm is used to identify the boundary of the key anatomical structure, and an initial feature set of the anatomical structure is generated; Based on the initial feature set of the anatomical structure, the spatial registration is performed between the real-time collected anatomical features and the pre-constructed standard anatomical model by using an iterative closest point algorithm, and an accurate anatomical structure model after registration is generated; In combination with the movement trajectory data of the surgical instrument, the relative position relationship between the instrument and the tissue in the accurate anatomical structure model after registration is dynamically updated, and a dynamic three-dimensional augmented reality model of the key anatomical structure in surgery is constructed; The dynamic three-dimensional augmented reality model is hierarchically decomposed, the static anatomical structure feature layer and the dynamic instrument interaction feature layer are separated, and a double-feature layer model representation is generated.

9. A storage medium, characterized by The storage medium stores a computer program, and the computer program is configured to execute the method in any one of claims 1-5 when running.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the computer program to execute the method in any one of claims 1-5.

Images