A virtual reality-based urological surgery assistance system and method of operation

Abstract

The application discloses a kind of based on virtual reality urological surgery auxiliary system and operating method, comprising: multi-modal data fusion processing module, for integrating patient CT / MRI image, ultrasonic real-time data and anatomical structure data, constructs urinary organ three-dimensional model;Virtual reality surgical environment construction module is used to generate immersive virtual surgery scene;Surgical instrument real-time tracking and mapping module is used to track physical surgical instrument pose in real time by spatial positioning device;Real-time physics engine and tissue deformation simulation module is used to simulate the interaction of surgical instrument and virtual tissue;Surgical navigation and early warning module is used to provide key structure highlight, safety boundary prompt and risk early warning based on preoperative planning and real-time data.The application realizes the full-process intelligentization, precision and immersive assistance of urological surgery from preoperative planning, intraoperative navigation to skill training, significantly improves the safety, efficiency and teaching effect of operation.

Description

Technical Field

[0001] This invention relates to the field of medical technology and virtual reality, and in particular to a virtual reality-based urological surgical assistance system and its operation method. Background Technology

[0002] Urological surgeries, especially laparoscopic, robot-assisted, or open surgeries involving deep organs such as the kidneys, prostate, and bladder, demand extremely high precision and safety. Surgeons must operate precisely within complex anatomical structures, avoiding damage to vital blood vessels and nerves while ensuring complete removal of the lesion. However, traditional surgeries, relying on the surgeon's experience and anatomical knowledge, as well as the assistance of intraoperative two-dimensional imaging (such as ultrasound), have the following limitations: (1) Insufficient spatial perception: It is difficult to construct accurate three-dimensional anatomical relationships based on two-dimensional images, especially when dealing with variant anatomy or complex tumors, which increases the risk of accidental injury.

[0003] (2) Lack of real-time navigation: During the operation, it is impossible to understand the precise positional relationship between surgical instruments and key structures (such as tumor boundaries, ureters, and neurovascular bundles) in real time and intuitively.

[0004] (3) Disconnect between preoperative planning and intraoperative planning: Exquisite preoperative three-dimensional reconstruction models often cannot be seamlessly integrated into the surgical procedure, thus limiting their guiding value.

[0005] (4) High training costs: The training of young physicians relies heavily on the "mentor-apprentice" model and animal experiments. The training cycle is long, the cost is high, and there is a lack of objective and quantitative evaluation methods.

[0006] These existing solutions generally suffer from the following problems: First, the model accuracy and realism are insufficient to support complex surgical decisions; second, they lack effective integration with real-time intraoperative data, and the models are static and cannot reflect tissue deformation; third, the systems are closed, failing to form a complete closed loop from planning and navigation to evaluation and archiving; and fourth, their simulation capabilities for the unique physical properties of urological organs and surgical procedures (such as delicate dissection, electrocoagulation, ureteral anastomosis, etc.) are weak.

[0007] Therefore, there is an urgent need in this field for a urological surgical assistance system that can deeply integrate multimodal data, provide a high degree of immersion and real-time interactivity, and has the ability to provide accurate navigation and objective evaluation. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by providing a virtual reality-based urological surgical assistance system. This system integrates high-precision 3D modeling, immersive VR rendering, real-time instrument tracking, biomechanical simulation, and intelligent navigation and early warning to provide comprehensive assistance throughout urological surgeries, thereby improving surgical precision, safety, and teaching efficiency.

[0009] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solution: a virtual reality-based urological surgical assistance system, comprising: The multimodal data fusion processing module is used to receive and fuse the patient's preoperative CT / MRI image data, intraoperative real-time ultrasound data, and standard urinary system anatomical atlas data. Through deep learning segmentation networks and three-dimensional reconstruction algorithms, it generates a personalized, high-precision three-dimensional model of the urinary system that includes tissues, organs, blood vessels, nerves, and lesions. A virtual reality surgical environment construction module, connected to the multimodal data fusion processing module, is used to construct an immersive virtual surgical scene based on the three-dimensional model. The scene supports multi-view observation, tissue transparency adjustment, and hierarchical anatomical display. The surgical instrument real-time tracking and mapping module is used to track the spatial position and attitude of physical surgical instruments in real time through an optical or electromagnetic spatial positioning system, and map them to the virtual surgical environment with high precision and low latency to generate corresponding virtual surgical instrument models. The physics engine and tissue deformation simulation module are connected to the virtual reality surgical environment construction module and the surgical instrument real-time tracking and mapping module. They are used to simulate the physical interaction between virtual surgical instruments and virtual tissues, and to calculate and visualize the cutting, coagulation, bleeding, traction deformation and physiological reactions of tissues in real time. The surgical navigation and early warning module is connected to the multimodal data fusion processing module and the physical engine and tissue deformation simulation module. It is used to load the preoperative planning path, display the positional relationship of surgical instruments relative to key anatomical structures in real time in the virtual environment, and trigger visual, auditory or tactile warnings when the instruments approach or exceed the safety boundary. The surgical skills assessment and recording module is used to collect the surgeon's operational data during surgical training or actual surgery, quantify the quality of the operation based on a predefined assessment index system, and generate a structured report.

[0010] Furthermore, the multimodal data fusion processing module specifically includes: The image data preprocessing unit is used to perform denoising, registration and interpolation on the input CT / MRI image sequences to obtain isotropic three-dimensional volume data; The deep learning segmentation unit uses a convolutional neural network based on the U-Net++ architecture to perform pixel-level semantic segmentation on the three-dimensional volume data, and identifies and labels key structures such as kidneys, ureters, bladders, prostates, tumors, arteries, veins and nerves. The 3D model reconstruction unit, based on the segmentation results, uses the moving cube algorithm to reconstruct the surface and generate a 3D model of triangular facets with topological structure. The multimodal registration and fusion unit is used to perform non-rigid registration between intraoperative real-time ultrasound images and the three-dimensional model, realizing dynamic alignment and updating of the real anatomical structure during surgery and the preoperative model. It also incorporates statistical information from standard anatomical atlases as prior knowledge into the model, enhancing the anatomical integrity and adaptability of the model.

[0011] Furthermore, the virtual reality surgical environment construction module specifically includes: The scene rendering engine uses real-time ray tracing technology to render the 3D model, simulating the organization's lighting, shadows, and material properties to provide a high-fidelity visual experience. The interactive interface unit provides a graphical user interface that allows users to switch perspectives, zoom and rotate models, control the visibility of layers, and access measurement tools via gestures, voice, or a VR controller. The scenario database stores various typical pathological models, standard surgical approach templates, and complication scenarios for teaching, training, and surgical rehearsals.

[0012] Furthermore, the real-time tracking and mapping module for surgical instruments specifically includes: The spatial positioning unit uses a high-precision optical infrared camera or electromagnetic sensor to capture real-time data from markers or sensors attached to physical surgical instruments and calculate the six-degree-of-freedom pose information of the instruments. The instrument model library stores three-dimensional geometric and kinematic models corresponding to various real urological surgical instruments. The coordinate mapping and calibration unit establishes a unified coordinate system between the real surgical space and the virtual environment, and uses a hand-eye calibration algorithm to accurately match the movement of physical instruments and virtual models, ensuring a mapping accuracy of less than 1 millimeter and a latency of less than 50 milliseconds.

[0013] Furthermore, the physics engine and tissue deformation simulation module specifically include: Biomechanical modeling unit, based on the finite element method or mass spring model, assigns viscoelastic and anisotropic biomechanical parameters to different virtual tissues; The real-time interactive computing unit receives instrument motion data from the surgical instrument real-time tracking and mapping module, calculates collision detection between the instrument and tissue, and dynamically simulates the effects of tissue deformation, cutting, electrocautery, suturing, and other operations based on a physical model. The physiological feedback simulation unit simulates the physiological reactions triggered by surgical procedures, including simulating ruptured blood vessels and bleeding, the effect of electrocoagulation hemostasis, ureteral peristalsis, and organ displacement caused by traction.

[0014] Furthermore, the surgical navigation and early warning module specifically includes: The preoperative planning import unit is used to import the surgical path, resection range, and key structures to be protected that have been pre-marked by the doctor on the 3D model; The real-time position sensing unit continuously calculates the shortest spatial distance between the tip of the virtual surgical instrument and key anatomical structures (such as tumor boundaries, major blood vessels, and nerve bundles); The multimodal early warning unit, when the shortest spatial distance is lower than the preset safety threshold, marks the risk area in the virtual environment with a bright halo, color change or semi-transparent warning box, and simultaneously triggers an auditory alarm or generates a tactile warning through a force feedback device; The augmented reality overlay unit can optionally overlay key navigation information (such as blood vessel direction and tumor outline) onto the surgeon's actual surgical field of view in an augmented reality manner.

[0015] Furthermore, the surgical skills assessment and recording module specifically includes: The data acquisition unit records multiple parameters during the surgical process in real time, including surgical time, instrument movement path length, speed, acceleration, tissue processing time, simulated blood loss, and number of accidental touches on key structures. The evaluation model unit constructs an evaluation model based on machine learning algorithms, analyzes the collected data, and outputs a comprehensive evaluation score on operational economy, stability, accuracy, and risk control capabilities. The report generation unit integrates the assessment results with key time point videos of the surgical procedure and 3D trajectory diagrams of instrument paths to automatically generate a structured skills assessment report for teaching feedback or surgical quality analysis.

[0016] Furthermore, the system also includes a cloud-based collaboration and case library module, which includes: The cloud storage and synchronization unit is used to encrypt and store anonymized patient 3D models, surgical plans, surgical procedure records, and evaluation results on a cloud server. The remote expert collaboration unit allows authorized experts to access the virtual surgical environment via the network and conduct remote consultations, pathway planning guidance, or real-time surgical observation and guidance in the form of virtual avatars. The surgical case database accumulates and stores typical and difficult surgical cases, supporting retrieval and learning by disease type, surgical procedure, and difficulty, and can be used for training young physicians and as a reference for optimizing surgical plans.

[0017] The present invention also provides a method for operating the system, comprising the following steps: S1: Data Acquisition and Model Building: Acquire preoperative medical imaging data of the target patient, and generate a personalized high-precision three-dimensional model of the urinary system through the multimodal data fusion processing module; S2: Virtual Environment Preparation: Import the 3D model into the virtual reality surgical environment construction module, configure the virtual surgical scene, and load preoperative planning data; S3: Instrument Tracking and Registration: Activate the real-time tracking and mapping module for surgical instruments to complete the spatial registration and virtual mapping of physical surgical instruments; S4: Immersive Surgical Interaction: The surgeon wears a VR headset to observe anatomical structures and operate physical instruments in a virtual environment. The physical engine and tissue deformation simulation module simulate the tissue deformation and physiological changes brought about by the surgical operation in real time. S5: Real-time navigation and early warning: The surgical navigation and early warning module provides real-time positional relationship information of key structures during the surgeon's operation and issues early warnings when risks may occur; S6: Skills assessment and recording: During surgical training or in a specific mode, the surgical skills assessment and recording module automatically records operational data and generates an assessment report; S7: Data archiving and sharing: Optionally, surgical-related data can be archived through the cloud collaboration and case library module, or remote collaboration can be initiated.

[0018] The beneficial effects of the above-described technical solution of the present invention are as follows: 1. This invention uses multimodal data fusion to construct a high-precision personalized three-dimensional model containing lesions, blood vessels, and nerves, and combines real-time tracking and mapping to provide surgeons with a three-dimensional anatomical understanding that surpasses the naked eye and traditional images.

[0019] 2. This invention utilizes high-performance VR rendering and a physics engine to create a highly realistic virtual surgical environment, supporting surgeons to perform natural and intuitive interactive operations and observe tissue dynamic reactions in real time, greatly improving the effectiveness of preoperative planning and skills training.

[0020] 3. This invention closely integrates preoperative planning with intraoperative operation, providing real-time, visualized navigation information and multimodal early warning, proactively preventing surgical risks, and is especially beneficial for handling complex surgeries and difficult cases.

[0021] 4. This invention can automatically record and analyze operational data, providing objective and quantitative skills assessments, and offering a scientific basis for standardized training of young physicians and continuous improvement of surgical quality. Through cloud collaboration and a case database, it enables remote sharing of expert resources and the accumulation and inheritance of valuable surgical experience, promoting the optimal allocation of medical resources and the overall improvement of medical standards. Attached Figure Description

[0022] Figure 1 This is a block diagram of the virtual reality urological surgery assistance system of the present invention; Figure 2This is a block diagram of the multimodal data fusion processing module of the present invention; Figure 3 This is a block diagram of the virtual reality surgical environment construction module of the present invention; Figure 4 This is a block diagram of the surgical instrument real-time tracking and mapping module of the present invention; Figure 5 This is a block diagram of the physics engine and tissue deformation simulation module of the present invention; Figure 6 This is a block diagram of the surgical navigation and early warning module of the present invention; Figure 7 This is a block diagram of the surgical skills assessment and recording module of the present invention. Detailed Implementation

[0023] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0024] Example 1: System Composition and Working Principle refer to Figure 1 This embodiment of a virtual reality-based urological surgical assistance system mainly includes the following modules: The system includes a multimodal data fusion processing module 100, a virtual reality surgical environment construction module 200, a surgical instrument real-time tracking and mapping module 300, a physics engine and tissue deformation simulation module 400, a surgical navigation and early warning module 500, a surgical skills assessment and recording module 600, and an optional cloud collaboration and case library module 700.

[0025] The multimodal data fusion processing module 100 is the foundation of this system. It receives DICOM format CT / MRI images from the patient. The image data preprocessing unit first performs isotropic resampling, non-uniformity correction, and noise filtering on the images. Subsequently, the deep learning segmentation unit loads a pre-trained U-Net++ model to automatically segment the preprocessed three-dimensional volume data. Taking partial nephrectomy as an example, the model needs to accurately segment the kidney contour, tumor boundary, renal artery, renal vein, and collecting system. The segmentation results are then processed by the three-dimensional model reconstruction unit to generate a triangular mesh model using the moving cube algorithm. To further improve the model's intraoperative applicability, the multimodal registration and fusion unit uses feature point matching and deformation field calculation to register the two-dimensional images acquired by the intraoperative ultrasound probe onto the three-dimensional model in real time, enabling real-time model updates and compensating for organ displacement caused by pneumoperitoneum or tissue traction.

[0026] The image data preprocessing unit 1001 is used to denoise, register and interpolate the input CT / MRI image sequence to obtain isotropic three-dimensional volume data. Specifically, (1) Data denoising: Non-local mean filtering or a denoising convolutional neural network based on deep learning, such as DnCNN, is used. This method can effectively suppress Gaussian noise and salt-and-pepper noise in the image, while preserving the edge and texture information of the tissue to the maximum extent, avoiding holes or blurred boundaries during subsequent segmentation. (2) Data registration: If the patient has both CT and MRI images, multimodal registration is required. Rigid registration based on mutual information is used for initial alignment, and then B-spline free deformation transformation is used for non-rigid registration to correct soft tissue deformation caused by patient position, respiratory movement, etc., to ensure that the anatomical structures in the two modalities correspond accurately in space. (3) Interpolation and isotropic resampling: The inter-slice resolution of the original image is usually lower than the intra-slice resolution. The Z-axis direction is resampled using a trilinear interpolation algorithm to generate isotropic 3D volume data with voxel dimensions of 0.5mm × 0.5mm × 0.5mm. This ensures that jagged "staircase effects" do not occur when cutting and rendering the model in any direction.

[0027] The deep learning segmentation unit 1002 employs a convolutional neural network based on the U-Net++ architecture to perform pixel-level semantic segmentation on the 3D volumetric data, identifying and labeling key structures such as the kidneys, ureters, bladder, prostate, tumors, arteries, veins, and nerves. Compared to the standard U-Net, the U-Net++ architecture, by introducing dense skip connections between the encoder and decoder, fuses features at more different scales, enabling more accurate segmentation of microstructures (such as small blood vessels and nerves) and tumors with blurred boundaries. The U-Net++ architecture training and optimization include: Dataset: The dataset was trained using over 1,000 CT / MRI images of the urinary system that have been precisely annotated by senior radiologists and urologists.

[0028] Loss function: A combination of Dice Loss and Focal Loss is used. Dice Loss directly optimizes the overlap of segmented regions, while Focal Loss focuses on solving class imbalance problems (such as tumor voxels being far fewer than normal tissue voxels).

[0029] Output: The network outputs a probability vector for each voxel, representing the probability that it belongs to a category such as "kidney," "tumor," or "renal artery." The final semantic segmentation map can be obtained using the ArgMax function.

[0030] The 3D model reconstruction unit 1003, based on the segmentation results, uses the moving cube algorithm to reconstruct the surface, generating a 3D model with triangular facets and a topological structure. The moving cube algorithm traverses each cube voxel in the 3D volume data, and generates a series of triangular facets based on the label values ​​of its eight vertices by looking up a table, thereby constructing isosurfaces.

[0031] The multimodal registration and fusion unit 1004 is used to perform non-rigid registration between intraoperative real-time ultrasound images and the three-dimensional model, achieving dynamic alignment and updating of the actual anatomical structures during surgery and the preoperative model. It also incorporates statistical information from standard anatomical atlases as prior knowledge into the model, enhancing its anatomical integrity and adaptability to variations. Specifically, ultrasound data acquisition involves installing optical positioning markers on the laparoscopic ultrasound probe, enabling real-time tracking.

[0032] The registration process includes: (1) Feature extraction: Extract feature points (such as vascular branch points and organ boundaries) from real-time ultrasound images, and extract similar features from the corresponding sections of the preoperative three-dimensional model.

[0033] (2) Deformation field calculation: The Demons algorithm or a deformable registration algorithm based on B-splines is used. This algorithm regards registration as a diffusion process, and through iterative calculation, the feature points of the ultrasound image are "diffused" to the feature points of the model, and finally a dense displacement vector field (deformation field) is generated.

[0034] (3) Model-driven: Apply this deformation field to the entire preoperative three-dimensional model to make it deform and displace synchronously with the real organ, thus realizing the transformation from "static model" to "dynamic digital twin".

[0035] Anatomical atlas fusion uses statistical probability models (such as the common course of neurovascular bundles) from standard anatomical atlases like the Visible Human Project or Zygote Body as prior knowledge, and incorporates them into personalized models through Bayesian inference to provide reliable anatomical navigation in areas with poor image quality or unclear structures.

[0036] The virtual reality surgical environment construction module 200 is responsible for presenting the model. The scene rendering engine uses ray tracing rendering technology such as NVIDIA RTX to set different materials and scattering parameters for different tissues such as the kidneys, fat, and blood vessels, generating scenes with realistic lighting and shadow effects. The interactive interface unit allows the surgeon to use a handheld VR controller for flight navigation, freely adjust the viewing angle, or use voice commands to hide the kidneys to display tissues in layers and focus on deep blood vessels. The virtual reality surgical environment construction module 200 specifically includes: Scene rendering engine 2001 uses real-time ray tracing technology to render the three-dimensional model, simulating the lighting, shadows and material properties of the tissue to provide a high-fidelity visual experience. The rendering technology uses real-time ray tracing. Compared with traditional rasterization, ray tracing can accurately simulate the reflection, refraction and soft shadow effects of light. In addition, physically based rendering materials are set for different tissues. (1) Kidney / Prostate: A subsurface scattering model is used to simulate the soft, translucent texture produced by light scattering inside the organ. (2) Adipose tissue: A high scattering coefficient and rough surface reflection are set. (3) Blood vessels: A smooth reflection model is used on the inner wall, and the absorption (red) and scattering of light by blood inside are simulated.

[0037] The interactive interface unit 2002 provides a graphical user interface that allows users to switch perspectives, zoom and rotate models, control the visibility of layers, and access measurement tools via gestures, voice, or a VR controller.

[0038] The scenario database 2003 stores various typical pathological models, standard surgical approach templates, and complication scenarios for teaching, training, and surgical rehearsals. It uses a relational database (such as MySQL) to store case metadata (disease type, surgical procedure, difficulty level), while a file system or object storage (such as AWS S3) is used to store large 3D models and surgical video files. The surgical instrument real-time tracking and mapping module 300 serves as a bridge connecting the virtual and real worlds. This embodiment employs the OptiTrack optical positioning system. Reflective markers are installed on surgical instruments (such as electric hooks and dissecting forceps). The spatial positioning unit uses multiple infrared cameras arranged around the surgical area to capture the positions of these markers at a frequency of 120Hz. The coordinate mapping and calibration unit, through pre-calibrated systems, transforms the captured three-dimensional coordinate data into the coordinate system of the virtual world and calls the corresponding virtual instrument model (such as a virtual electric hook) from the instrument model library to drive the synchronous movement of the virtual and real instruments.

[0039] The spatial positioning unit 3001 uses a high-precision optical infrared camera or electromagnetic sensor to capture the data of markers or sensors attached to physical surgical instruments in real time and calculate the six-degree-of-freedom pose information of the instruments. The positioning uses (1) Optical positioning system: OptiTrack or Vicon system. 8-12 high frame rate (above 120Hz) infrared cameras are arranged around the ceiling of the operating room. Special passive reflective ball markers are installed on the instruments. The camera emits infrared light, which is reflected by the markers. The system calculates the three-dimensional coordinates of each marker point by triangulation and then calculates the complete six-degree-of-freedom pose of the instrument based on at least three non-collinear marker points. (2) Electromagnetic positioning system: As a backup solution, it is used when direct optical vision cannot be achieved (such as when it is blocked by the doctor's body). The system generates a controllable electromagnetic field and calculates the position and direction of the instrument by detecting the induced current of the micro-sensor at the tip of the implanted instrument in the magnetic field. The disadvantage is that it is easily interfered with by metal equipment in the operating room.

[0040] The instrument model library 3002 stores 3D geometric and kinematic models corresponding to various real urological surgical instruments. Each real instrument is scanned and modeled with high precision using SolidWorks or Blender to ensure consistent geometry. For jointed instruments (such as laparoscopic forceps and scissors), their kinematic chains (e.g., Denavit-Hartenberg parameters) are established, defining the degrees of freedom and range of motion for each joint. When the system detects changes in the joint angles of the physical instrument, it can drive the virtual model to perform completely consistent movements in real time.

[0041] The coordinate mapping and calibration unit 3003 establishes a unified coordinate system between the real surgical space and the virtual environment. Through a hand-eye calibration algorithm, it precisely matches the movements of physical instruments and the virtual model, ensuring a mapping accuracy of less than 1 mm and a latency of less than 50 milliseconds. Hand-eye calibration uses a standard calibration rod with marked points. In the real space, the calibration rod sequentially touches a reference point with known coordinates; simultaneously, in the virtual space, the coordinates of the tip of the virtual calibration rod are recorded. By collecting multiple sets of corresponding point pairs, the optimal rotation and translation matrix from the positioning system coordinate system to the virtual world coordinate system is solved using singular value decomposition or quaternion methods.

[0042] The physics engine and tissue deformation simulation module 400 imbue virtual tissue with "life." The biomechanical modeling unit sets different Young's modulus and Poisson's ratio for the renal cortex, medulla, tumors, and blood vessels. When the virtual electrocautery hook contacts virtual kidney tissue, the real-time interactive computing unit performs real-time collision detection and deformation calculation based on a point-mass spring model, simulating tissue indentation and traction. When virtual electrocoagulation is performed, the physiological feedback simulation unit simulates tissue whitening, carbonization, and the occlusion of microvessels. If an artery is accidentally injured, it simulates a gushing blood effect until hemostasis is achieved using virtual electrocoagulation forceps. Among these: Biomechanical modeling unit 4001, based on the finite element method (FEM) or the mass-spring model, assigns viscoelastic and anisotropic biomechanical parameters to different virtual tissues. The FEM divides a 3D tissue mesh into thousands of tiny tetrahedral elements, assigning each element parameters (such as Young's modulus and Poisson's ratio) from a hyperelastic material model (e.g., Mooney-Rivlin or Ogden model). While FEM offers high accuracy, it is computationally intensive. The mass-spring model treats the vertices of the tissue surface mesh as "masses," connected by virtual "springs" and "dampers." This is a lightweight alternative that simulates the viscoelasticity and anisotropy of tissues by adjusting the spring stiffness and damping coefficient. This system employs a hybrid model: FEM is used for core regions requiring high-precision deformation (e.g., tumors), while an optimized mass-spring model is used for surrounding tissues.

[0043] The real-time interactive computing unit 4002 receives instrument motion data from the real-time tracking and mapping module of the surgical instruments, calculates the collision detection between the instruments and tissues, and dynamically simulates the effects of tissue deformation, cutting, electrocautery, suturing and other operations based on the physical model. A hierarchical bounding box tree is used for fast collision detection. First, a rough AABB bounding box test is performed to quickly eliminate non-intersecting objects, and then a precise triangular facet level detection is performed on objects that may collide. When a collision occurs, the force applied by the instrument is used as a boundary condition and substituted into the physical model for solution. For the mass spring model, a large linear equation system is solved, and the conjugate gradient method is used for fast iterative solution. The surgical effect simulation includes: (1) Cutting: When the stress of the virtual scissors or electric hook exceeds the tissue fracture limit, the mesh is dynamically redrawn, and the mass and spring are split along the cutting path. (2) Electrocautery: The material color of the tissue near the contact point is changed in real time (whitening, charring), and the thermal damage range of the surrounding tissue is calculated through the heat conduction model.

[0044] The physiological feedback simulation unit 4003 simulates the physiological responses triggered by surgical procedures, including simulating ruptured blood vessels and bleeding, the effects of electrocoagulation, ureteral peristalsis, and organ displacement due to traction. For example, a position-based dynamics method is used to simulate blood particles. When a blood vessel is "cut," a large number of blood particles are generated at the rupture site according to blood pressure parameters. These particles are constrained by gravity, viscosity, and collision, resulting in flow and splashing effects. During electrocoagulation, a coagulated model is generated at the bleeding point, and the generation of blood particles stops. The spontaneous peristaltic waveform is simulated by applying a periodic sinusoidal displacement field to specific nodes of the ureter model.

[0045] The Surgical Navigation and Early Warning Module 500 is a guardian of surgical safety. Before surgery, surgeons can outline the tumor resection line (safety boundary) on a 3D model. During surgery, the real-time position sensing unit continuously calculates the distance between the virtual electric hook tip and the main renal artery. Once this distance falls below a preset 3mm safety threshold, the multimodal early warning unit immediately renders a flashing red semi-transparent warning shield around the renal artery, while a voice prompt "Approaching Artery" sounds in the headset. If the system is connected to a force feedback device, the controller will generate slight vibrations, including: The preoperative planning import unit 5001 is used to import surgical paths, resection ranges, and critical structures to be protected that have been pre-marked by the surgeon on a 3D model. For example, in the VR planning interface, the surgeon uses a virtual "marking pen" (controlled by a VR controller) to directly draw safety boundaries (such as tumor resection lines) and critical protected structures (such as neurovascular bundles) on the 3D model. This data is saved as a structured JSON or XML file.

[0046] The real-time position sensing unit 5002 continuously calculates the shortest spatial distance between the tip of the virtual surgical instrument and key anatomical structures (such as tumor boundaries, major blood vessels, and nerve bundles). It continuously calculates the nearest point-to-point distance between the virtual instrument tip and the mesh model of the key structures using the GJK algorithm or the separating axis theorem.

[0047] The multimodal early warning unit 5003, when the shortest spatial distance is lower than a preset safety threshold, marks the risk area in the virtual environment with a bright halo, color change, or semi-transparent warning box, and simultaneously triggers an auditory alarm or generates a tactile warning through a force feedback device. This includes visual warnings: a semi-transparent, color-gradient geometry (such as a flashing sphere from yellow to red) is dynamically generated in the risk area. The shader program adjusts its color and transparency in real time according to the proximity. Auditory warnings: using 3D spatial audio technology, the pitch and rhythm of the alarm sound become more rapid as the risk level increases, and the sound source appears to come from the direction of the risk, providing a sense of spatial orientation. Tactile warnings: through a force feedback device connected to the system, when an instrument crosses the safety boundary, the device outputs a reverse pulse force or continuous resistance, simulating the real tactile sensation of "touching" something.

[0048] The augmented reality overlay unit 5004 can optionally overlay key navigation information (such as blood vessel direction and tumor outline) onto the surgeon's actual surgical field of view in an augmented reality manner. An optical perspective-based AR headset is used. First, the real surgical environment is modeled and located in real time using SLAM technology. Then, using the coordinate mapping method described in module 300, the virtual model (such as tumor outline and blood vessels) is precisely "pinned" to the real anatomical location, achieving a perfect fusion of virtual and reality.

[0049] The Surgical Skills Assessment and Recording Module 600 is the core of teaching and quality control. In training mode, the data acquisition unit records all operational data of the trainee during a renal tumor enucleation surgery. The assessment model unit analyzes the total operation time, total instrument path length (economic efficiency), peak instantaneous speed of the instrument (stability), number of accidental touches to the collecting system (safety), and simulated blood loss. Finally, the report generation unit generates a detailed assessment report, pointing out the strengths and areas for improvement. This includes: The data acquisition unit 6001 records multiple parameters during the surgical process in real time, including surgical time, instrument movement path length, speed, acceleration, tissue processing time, simulated blood loss, and number of accidental touches on key structures.

[0050] Evaluation model unit 6002, based on machine learning algorithms, constructs an evaluation model to analyze collected data and output a comprehensive evaluation score regarding operational economy, stability, accuracy, and risk control capabilities. Meaningful features are extracted from the raw data, such as: total instrument path length (economy), root mean square of acceleration (stability), and dwell time within 3mm of critical structures (risk control). Ensemble learning algorithms such as random forest or gradient boosting decision tree are employed. The model is trained using a large amount of expert surgical data as a standard, learning the complex nonlinear relationship between various features and surgical quality, and finally outputting a comprehensive score of 0-100 and various sub-scores.

[0051] The report generation unit 6003 integrates the assessment results with key time point videos of the surgical procedure and 3D instrument path diagrams to automatically generate a structured skills assessment report for teaching feedback or surgical quality analysis. The assessment score, 3D instrument path diagram (using different colors to represent speed), time point screenshots of key events (such as accidental blood vessel contact), and recorded operation video clips are automatically populated into a preset HTML or PDF report template to generate a detailed analysis document.

[0052] The cloud-based collaboration and case library module 700 expands the system's boundaries. After successfully performing a complex radical prostatectomy, the surgeon can upload anonymized models, surgical videos, and evaluation reports to the cloud-based case library, tagged as a "high-difficulty case with nerve preservation." Doctors from other hospitals can request to observe the procedure through the remote expert collaboration unit, replaying the entire surgical process in a virtual environment and learning key techniques.

[0053] The cloud storage and synchronization unit 7001 is used to encrypt and store anonymized patient 3D models, surgical plans, surgical procedure records, and evaluation results on a cloud server. The storage layer is built on Alibaba Cloud OSS or Azure Blob Storage, and all uploaded data is encrypted with AES-256 at the client end. The database records file metadata and access permissions.

[0054] The Remote Expert Collaboration Unit 7002 allows authorized experts to access a virtual surgical environment via the network and conduct remote consultations, pathway planning guidance, or real-time surgical observation and guidance in the form of a virtual avatar. The remote expert's virtual avatar (a simplified 3D model) enters the surgeon's VR scene, and the expert can provide real-time guidance through voice communication, laser pointer annotation, and other methods.

[0055] The surgical case database contains 7,003 cases, accumulating and storing typical and difficult surgical cases. It supports searching and learning by disease type, surgical procedure, and difficulty, and can be used for training young physicians and as a reference for optimizing surgical plans.

[0056] Example 2: Application in radical prostatectomy This embodiment uses laparoscopic radical prostatectomy in the field of urology as an example to illustrate in detail the specific application process of the system of the present invention in a real clinical scenario. This surgery places extremely high demands on preserving the neurovascular bundle that controls erectile function and the urethral sphincter that prevents urinary incontinence; any deviation may lead to a serious decline in the patient's postoperative quality of life.

[0057] S1: Data Acquisition and Personalized Model Building First, pelvic multiparameter magnetic resonance imaging (mp-MRI) data were acquired from the target patients. mp-MRI includes T2-weighted images, diffusion-weighted imaging (DWI), and dynamic contrast-enhanced imaging (DCE) sequences, which can clearly show the anatomical divisions of the prostate, the location and extent of the tumor (especially extracapsular invasion), and its adjacent relationships with the seminal vesicles, bladder neck, rectum, and neurovascular bundles.

[0058] Then, deep learning-based automatic segmentation is performed. A U-Net++ model specifically trained for the prostate region is loaded to automatically segment the mp-MRI data. The model accurately identifies and labels the following key structures: Target organs: central zone, peripheral zone, and anterior fibromuscular matrix of the prostate.

[0059] Lesions: Tumor lesions (highlighted on DWI and DCE sequences), and the tumor volume and distance from the capsule are automatically calculated.

[0060] Key surrounding structures: bilateral neurovascular bundles, seminal vesicles, bladder, anterior rectal wall, and membranous urethra.

[0061] Using the moving cube algorithm, a 3D model containing all the above structures is generated. In the model, different tissues are distinguished by different colors; for example, the prostate is translucent amber, the tumor is red, the neurovascular bundle is bright yellow, and the rectum is brown.

[0062] The model incorporates statistical information from standard male pelvic anatomy atlases. This is particularly important for patients with anatomical variations, as the system can predict the most likely pathways of neurovascular bundles in areas where images are unclear, providing surgeons with additional safety.

[0063] S2: Virtual Reality Surgical Environment Preparation and Preoperative Planning The personalized 3D prostate model generated in S1 is imported into the virtual environment. Using real-time ray tracing technology, an immersive scene with realistic lighting and tissue texture (such as the glandular texture of the prostate and the gloss of blood vessels) is rendered.

[0064] The surgeon wears a VR headset and holds a VR controller, entering a virtual surgical environment. The surgeon can freely "fly" to any viewing angle using gestures, such as observing the relationship between the tumor and the urethra from directly above the pelvis, the side, or even inside the bladder. Tissue transparency can be adjusted, for example, "hiding" the bladder and rectum, allowing the surgeon to focus on the narrow surgical plane between the prostate and the neurovascular bundle.

[0065] The doctor uses the virtual "marker" function to make the following plans on the 3D model: Resection Boundary Marking: On the surface of the prostate, the planned resection boundary is precisely delineated around the tumor. The system automatically recommends a safe boundary of 2-3 mm based on the tumor's invasiveness. Doctors can then make fine adjustments to this boundary, striving to find the optimal balance between radical tumor removal and tissue preservation.

[0066] Key protective structures highlighted: The bilateral neurovascular bundles are marked as the highest level of "protective structures". Doctors can set a safety threshold of 1-2 mm away from the neurovascular bundles; any instrument entering this area will trigger a strong warning.

[0067] Surgical approach planning: Plan the general dissection sequence, such as first dealing with the bladder neck, then dissecting the seminal vesicles, and finally performing detailed dissection next to the neurovascular bundle.

[0068] S3: Real-time tracking and registration of surgical instruments Complete this before the surgery begins to ensure the virtual environment is synchronized with the real world. Attach optical positioning reflective markers to key laparoscopic instruments to be used in the surgery (such as monopolar electric hook scissors, bipolar electrocoagulation forceps, and non-traumatic grasping forceps).

[0069] A unified coordinate system is established between the real surgical space (using the laparoscopic camera as a reference) and the virtual environment using a hand-eye calibration algorithm. The surgeon uses a marked calibration rod to sequentially touch a reference point fixed on the patient's body surface to complete precise coordinate mapping. A virtual model that is completely identical to the real instrument (such as a virtual single-pole scissors) is retrieved and bound to the physical instrument. Finally, the surgeon slightly moves the real instrument and observes in the VR headset whether the virtual instrument achieves high-precision (error <1mm) and low-latency (latency <50ms) synchronized movement.

[0070] S4: Immersive Surgical Interaction and Real-time Deformation Simulation During the surgery, this system provides two interaction modes: Mode 1: VR Full Immersion Mode (for training or rehearsing key steps): The surgeon is completely immersed in the virtual environment, operating physical instruments, with all visual feedback coming from the VR headset. When using virtual monopolar scissors for tissue separation, the process of tissue traction and cutting is simulated in real time. For example, if a small blood vessel is encountered, bleeding will be simulated, and the surgeon must immediately switch to virtual bipolar electrocautery forceps for hemostasis. The tissue will be simulated to turn white and carbonize in real time.

[0071] Mode 2: Augmented Reality Overlay Mode (for actual surgery): The surgeon mainly views the real video footage from the laparoscopy, while key navigation information is overlaid onto the real field of vision through an AR headset or monitor.

[0072] When the surgeon uses monopolar scissors to dissect the apex of the prostate, he can not only see the real anatomical structure, but also the semi-transparent outline of the tumor and a virtual model of the urethral sphincter superimposed on the real tissue. This allows him to be more informed when severing the urethra and avoid damage.

[0073] When the prostate is stretched, the virtual model deforms in real time through a physics engine, predicting and displaying the displacement of adjacent organs such as the bladder and rectum, reminding the surgeon to pay attention to the force applied.

[0074] S5: Real-time navigation and multimodal early warning First, calculate the shortest spatial distance between the virtual surgical instrument (such as the tip of an electrocautery hook) and pre-defined key structures (such as neurovascular bundles and the rectal wall). For example, when dissecting the "danger zone" adjacent to the neurovascular bundle on the posterolateral side of the prostate, the following warning is issued: Visual Warning: In the VR or AR field of view, the neurovascular bundle is always surrounded by a soft, bright halo. When the distance between the device tip and the neurovascular bundle is less than the preset 2mm safety threshold, the color of the halo immediately changes from a safe green to a warning yellow, then to a dangerous red, and the brightness increases sharply, while the flashing frequency accelerates.

[0075] Auditory warning: Simultaneously, a voice warning will sound in the surgeon's headphones, "Caution, approaching the neurovascular bundle," with the tone becoming more urgent as the risk increases.

[0076] Tactile warning (if equipped): If the operator uses a force feedback device, when the instrument is about to enter a restricted area, the hand controller will generate a pulse vibration or gradually increasing resistance, just like touching an invisible "barrier" in reality, providing the most intuitive tactile warning.

[0077] The system will always display the pre-planned resection boundaries in the surgical field, guiding the surgeon to operate in the correct plane and ensuring complete tumor resection.

[0078] S6: Surgical Skills Assessment and Recording During training or postoperative review, record all of the surgeon's operations in detail: total surgical time, time the instrument spends near the neurovascular bundle, the economy of the instrument's movement path (total length), instantaneous speed (to assess stability), number of accidental touches to the neurovascular bundle and rectum, simulated bleeding volume, etc.

[0079] Then, based on machine learning models, this data is analyzed to generate a quantitative evaluation report. The report not only provides an overall score but also evaluates the operator's operational precision (the degree of adherence to the planned boundaries), stability (whether the movements are smooth), risk control ability (the effectiveness of protecting critical structures), and efficiency in each category.

[0080] The report integrates assessment scores with key time-point videos of the surgical procedure and instrument trajectory diagrams in three-dimensional space to automatically generate a structured surgical report. This report can be used for training feedback for young physicians or as an objective basis for continuous improvement of surgical quality.

[0081] S7: Data Archiving and Cloud Collaboration After the surgery, the anonymized 3D model of the patient, surgical plan, complete process record and evaluation report are encrypted and uploaded to the cloud server.

[0082] To verify the effectiveness of this system, a controlled study was conducted in the urology department of a large teaching hospital. Twenty resident physicians were randomly divided into two groups: the experimental group used the system for laparoscopic partial nephrectomy simulation training, while the control group used a traditional box simulator. After four weeks of training, five times a week, two senior experts conducted a blind evaluation of the two groups' performance in live animal surgeries (using the OSATS scale). Simultaneously, the system recorded objective operational data.

[0083] The results showed that the experimental group physicians significantly outperformed the control group in terms of OSATS total score, tissue processing efficiency, and avoidance of damage to important structures (p<0.05). The experimental group physicians also experienced a 15% reduction in the average total instrument path length, a 22% reduction in unnecessary movements, and a 30% reduction in simulated intraoperative bleeding. This demonstrates that the system can effectively accelerate the development of surgical skills and improve the economy and safety of the procedure.

[0084] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A virtual reality-based urological surgical assistance system, characterized in that, include: The multimodal data fusion processing module is used to receive and fuse the patient's preoperative CT / MRI image data, intraoperative real-time ultrasound data, and standard urinary system anatomical atlas data. Through deep learning segmentation networks and three-dimensional reconstruction algorithms, it generates a three-dimensional model of the urinary system that includes tissues, organs, blood vessels, nerves, and lesions. The virtual reality surgical environment construction module is used to construct an immersive virtual surgical scene based on the three-dimensional model of the urinary system. The scene supports multi-view observation, tissue transparency adjustment, and hierarchical anatomical display. The surgical instrument real-time tracking and mapping module is used to track the spatial position and attitude of physical surgical instruments in real time through an optical or electromagnetic spatial positioning system, and map them to the virtual surgical environment to generate corresponding virtual surgical instrument models. The physics engine and tissue deformation simulation module are used to simulate the physical interaction between virtual surgical instruments and virtual tissues, and to calculate and visualize the cutting, coagulation, bleeding, traction deformation and physiological response of tissues in real time. The surgical navigation and early warning module is used to load the preoperative planning path, display the positional relationship of surgical instruments relative to key anatomical structures in real time in the virtual environment, and trigger visual, auditory or tactile warnings when the instruments approach or exceed the safety boundary. The surgical skills assessment and recording module is used to collect the surgeon's operational data during surgical training or actual surgery, quantify the quality of the operation based on a predefined assessment index system, and generate a structured report.

2. The urological surgical assistance system based on virtual reality according to claim 1, characterized in that, The multimodal data fusion processing module specifically includes: The image data preprocessing unit is used to perform denoising, registration and interpolation on the input CT / MRI image sequences to obtain isotropic three-dimensional volume data; The deep learning segmentation unit uses a convolutional neural network based on the U-Net++ architecture to perform pixel-level semantic segmentation on the three-dimensional volume data, and identifies and labels key structures, including but not limited to the kidney, ureter, bladder, prostate, tumor, artery, vein and nerve. The 3D model reconstruction unit, based on the segmentation results, uses the moving cube algorithm to reconstruct the surface and generate a 3D model of triangular facets with topological structure. The multimodal registration and fusion unit is used to perform non-rigid registration of intraoperative real-time ultrasound images with the three-dimensional model, and to incorporate statistical information from standard anatomical atlases as prior knowledge into the three-dimensional model.

3. The urological surgical assistance system based on virtual reality according to claim 1, characterized in that, The virtual reality surgical environment construction module specifically includes: The scene rendering engine uses real-time ray tracing technology to render the 3D model and simulate the organization's lighting, shadows, and material properties. The interactive interface unit provides a graphical user interface that allows users to switch perspectives, zoom and rotate models, control the visibility of layers, and access measurement tools via gestures, voice, or a VR controller. The scenario database stores various typical pathological models, standard surgical approach templates, and complication scenarios for teaching, training, and surgical rehearsals.

4. The urological surgical assistance system based on virtual reality according to claim 1, characterized in that, The real-time tracking and mapping module for surgical instruments specifically includes: The spatial positioning unit uses an optical infrared camera or electromagnetic sensor to capture real-time data from markers or sensors attached to physical surgical instruments and calculate the six-degree-of-freedom pose information of the instruments. The instrument model library stores three-dimensional geometric and kinematic models corresponding to various real urological surgical instruments. The coordinate mapping and calibration unit establishes a unified coordinate system between the real surgical space and the virtual environment, and matches the movement of physical instruments and virtual models through a hand-eye calibration algorithm.

5. A virtual reality-based urological surgical assistance system according to claim 1, characterized in that, The physics engine and tissue deformation simulation module specifically include: Biomechanical modeling unit, based on the finite element method or mass spring model, assigns viscoelastic and anisotropic biomechanical parameters to different virtual tissues; The real-time interactive computing unit receives instrument motion data from the surgical instrument real-time tracking and mapping module, calculates collision detection between the instrument and tissue, and dynamically simulates the effects of operations on tissue, including but not limited to deformation, cutting, electrocautery, and suturing, based on a physical model. The physiological feedback simulation unit simulates the physiological reactions triggered by surgical procedures, including simulating ruptured blood vessels and bleeding, the effect of electrocoagulation hemostasis, ureteral peristalsis, and organ displacement caused by traction.

6. A virtual reality-based urological surgical assistance system according to claim 1, characterized in that, The surgical navigation and early warning module specifically includes: The preoperative planning import unit is used to import the surgical path, resection range, and key structures to be protected that have been pre-marked by the doctor on the 3D model; The real-time position sensing unit continuously calculates the shortest spatial distance between the tip of the virtual surgical instrument and key anatomical structures. The multimodal early warning unit, when the shortest spatial distance is lower than the preset safety threshold, marks the risk area in the virtual environment with a bright halo, color change or semi-transparent warning box, and simultaneously triggers an auditory alarm or generates a tactile warning through a force feedback device; Augmented reality overlay unit overlays key navigation information onto the surgeon's actual surgical field of view in an augmented reality manner.

7. A virtual reality-based urological surgical assistance system according to claim 1, characterized in that, The surgical skills assessment and recording module specifically includes: The data acquisition unit records multiple parameters during the surgical process in real time, including surgical time, instrument movement path length, speed, acceleration, tissue processing time, simulated blood loss, and number of accidental touches on key structures. The evaluation model unit constructs an evaluation model based on machine learning algorithms, analyzes the collected data, and outputs a comprehensive evaluation score on the quality of operation. The report generation unit integrates the assessment results with key time point videos of the surgical procedure and 3D trajectory diagrams of instrument paths to automatically generate a structured skills assessment report for teaching feedback or surgical quality analysis.

8. A virtual reality-based urological surgical assistance system according to claim 2, characterized in that, The system also includes a cloud-based collaboration and case library module, which includes: The cloud storage and synchronization unit is used to encrypt and store anonymized patient 3D models, surgical plans, surgical procedure records, and evaluation results on a cloud server. The remote expert collaboration unit allows authorized experts to access the virtual surgical environment via the network and conduct remote consultations, pathway planning guidance, or real-time surgical observation and guidance in the form of virtual avatars. The surgical case database accumulates and stores typical and difficult surgical cases, supporting retrieval and learning by disease type, surgical procedure, and difficulty, and can be used for training young physicians and as a reference for optimizing surgical plans.

9. The method of operating a virtual reality-based urological surgical assistance system according to any one of claims 1-8, characterized in that, Includes the following steps: S1: Data Acquisition and Model Building: Acquire preoperative medical imaging data of the target patient and generate a three-dimensional model of the urinary system through the multimodal data fusion processing module; S2: Virtual Environment Preparation: Import the 3D model of the urinary system into the virtual reality surgical environment construction module, configure the virtual surgical scene, and load the preoperative planning data; S3: Instrument Tracking and Registration: Activate the real-time tracking and mapping module for surgical instruments to complete the spatial registration and virtual mapping of physical surgical instruments; S4: Immersive Surgical Interaction: The surgeon wears a VR headset to observe anatomical structures and operate physical instruments in a virtual environment. The physical engine and tissue deformation simulation module simulate the tissue deformation and physiological changes brought about by the surgical operation in real time. S5: Real-time navigation and early warning: The surgical navigation and early warning module provides real-time positional relationship information of key structures during the surgeon's operation and issues early warnings when risks may occur; S6: Skills assessment and recording: During surgical training or in a specific mode, the surgical skills assessment and recording module automatically records operational data and generates an assessment report; S7: Data archiving and sharing: Archive surgical-related data through the cloud collaboration and case library module, or initiate remote collaboration.

Images