Surgical simulation methods and systems based on 3D image reconstruction and surgical navigation

Abstract

This surgical simulation method and system, based on 3D image reconstruction and surgical navigation, relates to the fields of biomedical engineering and virtual reality processing software. The conversion of patient medical image data into a 3D model provides a precise spatial information foundation for surgical simulation and generates multiple optional surgical navigation schemes, fully considering the feasibility of different incision locations. During virtual reality simulated surgery, dynamic display based on the user's actual perspective greatly enhances the realism and immersion of the simulation. When an operation fails, not only can the entire process of the failed operation be replayed, but alternative solutions can also be intelligently recommended, significantly improving the surgeon's learning efficiency and surgical skills. By combining image reconstruction, multi-scheme navigation, and intelligent feedback, the accuracy, flexibility, and effectiveness of surgical simulation are greatly improved.

Description

Technical Field

[0001] This application relates to the fields of biomedical engineering and virtual reality processing software, and in particular to surgical simulation methods and systems based on image 3D reconstruction and surgical navigation. Background Technology

[0002] With the continuous advancement of medical technology, surgical simulation has become widely used in the medical field as an important preoperative preparation and training tool. Surgical simulation helps doctors better understand patients' conditions, develop appropriate surgical plans, and improve surgical success rates and safety.

[0003] A common surgical simulation method is to plan surgery based on the patient's medical imaging data. Doctors analyze the patient's CT or MRI images, combined with their own experience, to determine the location and extent of the lesion and formulate a preliminary surgical plan.

[0004] In practice, doctors need to construct a three-dimensional image of the patient's internal structures in their minds based on images, which requires a high level of spatial imagination and experience. Especially when dealing with complex cases, if doctors cannot accurately grasp the precise spatial relationship between the lesion and surrounding important tissues, the risk of intraoperative misjudgment increases. For example, in neurosurgery, even millimeter-level errors can lead to serious consequences. Summary of the Invention

[0005] This application provides a surgical simulation method and system based on image 3D reconstruction and surgical navigation, which enables doctors to fully understand the 3D structure of the lesion site before surgery and obtain targeted operation training, thereby improving the effectiveness of surgical simulation.

[0006] In a first aspect, this application provides a surgical simulation method based on image 3D reconstruction and surgical navigation, comprising: generating a 3D model of the lesion site based on patient medical image data through image segmentation and 3D reconstruction; determining multiple optional surgical navigation schemes, including different incision locations, based on the 3D model of the lesion site, patient anatomical structure data, and the location of important organs; performing virtual reality simulated surgery according to the first surgical navigation scheme selected by the user, and displaying the surgical field of view according to the user's actual perspective based on the user's operation instructions during the simulated surgery; in the event of a failed simulated surgical operation, displaying the entire 3D field of view of the first operation that caused the surgical failure, and resetting the simulated surgical progress to before the first operation; and if the number of failed simulated surgical operations for the first operation exceeds a preset failure limit, displaying a recommended second surgical navigation scheme, wherein the second surgical navigation scheme is a surgical navigation scheme among the multiple optional surgical navigation schemes that does not include the first operation.

[0007] By employing the aforementioned technical solution, the transformation from patient medical imaging data to a 3D model was achieved, providing a precise spatial information foundation for surgical simulation. Based on this, the system can generate multiple optional surgical navigation plans, fully considering the feasibility of different incision locations. During virtual reality simulated surgery, the system dynamically displays information according to the user's actual perspective, greatly enhancing the realism and immersion of the simulation. When an operation fails, the system can not only replay the entire failed operation but also intelligently recommend alternative solutions. This feedback and optimization mechanism significantly improves the surgeon's learning efficiency and surgical skills. Overall, this method, by combining image reconstruction, multi-plan navigation, and intelligent feedback, greatly enhances the accuracy, flexibility, and effectiveness of surgical simulation.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, the determination of multiple optional surgical navigation schemes, including different incision locations, based on the three-dimensional model of the lesion site, patient anatomical data, and the location of important organs, specifically includes: determining multiple possible incision locations on the surface of the three-dimensional anatomical model representing the patient as a whole, according to preset safety zone rules; designing one or more paths for each incision location that avoid the location of important organs and reach the lesion site in the three-dimensional model of the lesion site based on a path planning algorithm; calculating the length, curvature, and distance to the location of important organs for each path based on the patient's anatomical data, and obtaining a path score according to a preset scoring standard; and identifying multiple paths that exceed a preset scoring threshold as the multiple optional surgical navigation schemes.

[0009] By employing the aforementioned technical solution, the system can accurately locate multiple possible incision sites on the surface of the patient's three-dimensional anatomical model based on preset safety zone rules. The application of path planning algorithms ensures that each incision site has one or more safe paths to the lesion site, avoiding vital organs. By calculating the path length, curvature, and distance to vital organs, and evaluating them according to preset scoring criteria, the system can select the optimal surgical navigation plan. This method not only considers surgical safety but also optimizes surgical efficiency and accuracy. By automatically generating and evaluating multiple surgical plans, this technology significantly reduces the workload of surgeons in the preoperative planning stage while improving the scientific rigor and feasibility of surgical plans.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the display of the surgical field of view according to the user's actual perspective specifically includes: real-time acquisition of the user's head position, rotation angle, or eye movement data to obtain perspective information; wherein, the perspective information refers to the user's observation position and observation direction in the virtual environment, the observation position being determined by the head position, and the observation direction being determined by the rotation angle or eye movement data; updating virtual camera parameters based on the perspective information, the virtual camera parameters including the position and orientation of the camera in the virtual environment; wherein, the virtual camera refers to the observation point that simulates a real camera in the three-dimensional virtual environment; and re-rendering the surgical scene using the updated virtual camera parameters and pushing it for display.

[0011] By employing the aforementioned technical solution, the system can capture the user's head position, rotation angle, or eye movement data in real time, thereby accurately obtaining the user's observation position and direction in the virtual environment. This data is used to update the virtual camera parameters, ensuring that the perspective in the virtual environment remains synchronized with the user's actual perspective. By rendering and pushing updated surgical scenes in real time, the system creates a highly realistic visual experience. This dynamic perspective adjustment technology not only enhances the realism of simulated surgery but also allows users to observe and operate the virtual surgical environment in the most natural way. This greatly enhances the effectiveness of simulation training, enabling doctors to better adapt to the real surgical environment and improve their surgical skills.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, displaying the entire three-dimensional view of the first operation that led to the surgical failure in the event of a simulated surgical operation failure specifically includes: if the surgical operation is determined to have failed based on preset evaluation criteria, tracing back the entire surgical process to determine the first operation that led to the surgical failure; calling up the complete three-dimensional record of the first operation, including the operator's perspective, hand movements, and instrument trajectory; generating the entire three-dimensional view of the first operation based on the complete three-dimensional record of the first operation, and displaying it on the user interface.

[0013] By employing the aforementioned technical solutions, the system can quickly pinpoint the critical steps leading to surgical failure. By reviewing the entire surgical procedure and accessing complete 3D recordings, the system can accurately reproduce the entire failed operation, including the operator's perspective, hand movements, and instrument trajectories. This comprehensive 3D playback not only allows users to intuitively understand the cause of the failure but also provides rich visual information for in-depth analysis and improvement. In this way, the system significantly improves learning efficiency, enabling doctors to learn from failures and rapidly improve their techniques. This immediate and detailed feedback mechanism greatly enhances the educational value and practicality of surgical simulation.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: after the user selects a next surgical navigation scheme from the recommended second surgical navigation schemes, displaying a comparison view including the differences between the next surgical navigation scheme and the first surgical navigation scheme; and resetting the virtual reality simulated surgery based on the next surgical navigation scheme.

[0015] By adopting the above technical solution, the system provides an intuitive comparative view when recommending new surgical navigation plans, clearly demonstrating the differences between the old and new plans. This visual comparison allows doctors to quickly understand the advantages and disadvantages of different plans, providing strong support for decision-making. Simultaneously, the system can reset the virtual reality simulation surgery based on the newly selected plan, allowing doctors to immediately try new surgical paths. This flexible switching and comparison mechanism greatly improves the efficiency and accuracy of surgical plan selection, enabling doctors to more comprehensively evaluate different surgical strategies and thus choose the optimal plan. This not only optimizes the preoperative planning process but also improves the quality of the final surgical plan.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the solution further includes: when a user selects a surgical navigation scheme for virtual reality simulated surgery, adjusting the list of assistant surgeons during the simulated surgery based on the user's modified operations; during the simulated surgery, automatically matching and completing the corresponding operations of the assistant surgeons based on the historical operation records of the assistant surgeons in the historical surgical database.

[0017] By adopting the above technical solution, the system allows users to flexibly adjust the list of assistant surgeons in simulated surgery, increasing the realism and adaptability of the simulation. More importantly, the system can automatically match and complete the corresponding operations of the assistant surgeons based on records in the historical surgical database. This intelligent auxiliary function not only improves the integrity and realism of the simulated surgery, but also enables the lead surgeon to better rehearse and adapt to the cooperation modes of different support teams. By simulating a real team collaboration environment, this technology significantly enhances the practical value of surgical simulation and helps improve the coordination ability and overall efficiency of the entire surgical team.

[0018] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: receiving multiple combinations of assistant surgeons in a hospital input by a user; determining the success rate of each assistant surgeon combination participating in each optional surgical navigation scheme based on the historical surgical database; and, when displaying a recommended second surgical navigation scheme, ranking the surgical navigation schemes according to the patient's planned surgery time, the time schedule of each assistant surgeon combination in the current hospital, and the success rate.

[0019] By employing the aforementioned technical solution, the system can comprehensively consider multiple key factors, including the historical success rates of different combinations of surgical assistants, the patient's planned surgery time, and the hospital's current staffing schedule. Using this data, the system can intelligently rank recommended surgical navigation options. This multi-dimensional evaluation and ranking mechanism not only considers the technical feasibility of the surgery itself but also fully takes into account the actual situation of hospital resources. Such intelligent recommendations greatly improve the efficiency and accuracy of surgical planning, helping hospitals optimize resource allocation and increase overall surgical success rates. Simultaneously, it provides physicians with more comprehensive decision support, aiding in the selection of the optimal surgical plan and team combination.

[0020] In a second aspect, this application provides a surgical simulation system comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the surgical simulation system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, this application provides a computer program product containing instructions that, when run on a surgical simulation system, cause the surgical simulation system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a surgical simulation system, cause the surgical simulation system to perform the method described in the first aspect and any possible implementation thereof.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By employing 3D reconstruction technology based on medical image data, multi-scheme surgical navigation generation algorithms, and intelligent failure feedback and optimization mechanisms, the technology effectively solves the technical problems of lack of accurate spatial information, single surgical scheme, and low learning efficiency in existing surgical simulations. This results in high-precision, multi-scheme, and intelligent feedback surgical simulation effects, significantly improving the accuracy, flexibility, and teaching effectiveness of surgical simulations.

[0025] 2. By adopting automatic surgical failure identification technology based on preset evaluation standards, a three-dimensional surgical backtracking mechanism for the entire process, and precise reproduction and visualization technology of failed operations, the technology effectively solves the technical problems of untimely surgical simulation feedback, unintuitive failure cause analysis, and low learning efficiency in existing technologies. This enables real-time, comprehensive, and intuitive surgical failure analysis and learning effects, significantly improving the educational value and practicality of surgical simulation.

[0026] 3. Due to the adoption of adjustable assistant surgeon configuration function and intelligent auxiliary operation matching technology based on historical surgical database, the technical problems of lack of team collaboration environment and unrealistic auxiliary operation in existing surgical simulation are effectively solved. Thus, a highly realistic surgical team collaboration simulation effect is achieved, which significantly enhances the practical value of surgical simulation and the training effect of team coordination ability. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a surgical simulation method based on image 3D reconstruction and surgical navigation in an embodiment of this application;

[0028] Figure 2 This is another flowchart illustrating the surgical simulation method based on image 3D reconstruction and surgical navigation in the embodiments of this application;

[0029] Figure 3 This is an exemplary hardware structure diagram of the surgical simulation system in the embodiments of this application. Detailed Implementation

[0030] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0031] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0032] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0033] (1) Image segmentation:

[0034] Image segmentation is the process of decomposing a digital image into multiple regions with similar characteristics, so that each region corresponds to an object or structure in the image. It is a fundamental technique in image analysis and understanding.

[0035] The goal of image segmentation is to simplify or alter the representation of an image, making it more meaningful and easier to analyze. It decomposes an image into a set of disjoint regions, each typically corresponding to an object or structure within the image. Pixels in the image are grouped together according to certain criteria to form a segmented region.

[0036] The technical solution of this application requires extracting the lesion tissue region based on the patient's medical image data, such as CT or MRI scans, using image segmentation technology, and then constructing a three-dimensional model of the lesion site using the segmentation results. For example, threshold segmentation, edge detection, and other algorithms can be used to process the scan images, identify the contours of lesions such as tumors, and then segment them from the background normal tissue to generate a three-dimensional model of the lesion site. This provides important basic data for subsequent surgical simulation and navigation.

[0037] (2) Three-dimensional reconstruction:

[0038] 3D reconstruction refers to the process of recovering the shape and internal structure of a 3D object from a 2D tomographic image using image processing technology.

[0039] 3D reconstruction technology analyzes structural information in 2D images to establish relationships between them, thereby reconstructing a 3D scene. In the medical field, it can reconstruct 3D digital models of human tissues and organs using 2D tomographic images such as X-rays, CT scans, and MRI. 3D reconstruction provides doctors with an interactive 3D view, aiding in lesion localization, surgical planning, and other tasks.

[0040] The technical solution of this application requires generating a three-dimensional digital model of the lesion (such as a tumor) based on the patient's CT or MRI scan images using three-dimensional reconstruction technology. First, the contour of the lesion tissue is extracted through image segmentation; then, based on the image capture parameters, segmentation results, and the positional relationship of adjacent slices, corresponding points in each image layer are determined to restore the three-dimensional shape; finally, the generated three-dimensional digital model is used to establish a virtual scene of the lesion, providing data support for subsequent surgical simulation and navigation. This method can accurately display the three-dimensional location, size, and shape of the lesion, which is beneficial for improving the effectiveness of surgical planning.

[0041] (3) Incision location:

[0042] The incision location refers to the position chosen externally to make an incision inside the body during surgery. Choosing the correct incision location is crucial for successful surgery, as it determines whether the target lesion can be easily reached and whether damage to vital organs and blood vessels can be avoided. The selection of the incision location requires consideration of multiple factors, including the location of the lesion and the surgical approach. A good incision location provides a clear surgical field, ensuring a smooth and safe procedure.

[0043] In the technical solution of this application, multiple optional incision locations need to be determined based on the three-dimensional model of the lesion site and the patient's anatomical structure data, serving as different surgical navigation options. For example, for intracranial tumors, a parietal bone incision or an occipital bone incision can be selected; for abdominal tumors, a midline incision or a perforation incision can be selected. The optimal incision can be selected according to the location of the lesion and the surgical requirements. This allows for pre-planning of the incision, targeted surgical simulation training, and improved surgical outcomes.

[0044] (4) Surgical navigation plan:

[0045] A surgical navigation plan is a pre-defined plan outlining the navigation path and surgical steps for a specific procedure. Based on the location and nature of the lesion, and considering the patient's individual anatomical characteristics, the surgical navigation plan pre-plans each step of the surgery, including incision location, incision path, and specific operational sequence. A good surgical navigation plan can minimize the need to avoid important blood vessels and nerves, allowing for direct and safe access to the lesion for treatment.

[0046] In the technical solution of this application, multiple optional surgical navigation plans need to be determined based on the three-dimensional model of the lesion and anatomical structure data, each plan corresponding to a different incision location. For example, renal cell carcinoma resection surgery can have laparoscopic incision plans and open incision plans. Each plan includes the precise incision location, the path to the tumor, precautions, etc. Surgeons can select the optimal plan based on the simulation results, conduct targeted training, master various possible surgical situations, and improve the success rate of surgery.

[0047] (5) Virtual Reality Simulated Surgery:

[0048] Virtual reality (VR) surgical simulation utilizes VR technology to simulate surgical procedures in a virtual environment. Doctors can perform simulated surgeries using the system, which provides real-time feedback based on various situations. This allows doctors to familiarize themselves with the surgical procedure beforehand, assess surgical plans, and improve their surgical skills.

[0049] In the technical solution of this application, a virtual surgical scene containing a three-dimensional model of the lesion can be constructed, and simulated surgery can be performed according to the selected surgical navigation scheme. Doctors can wear VR devices or directly view the first-person perspective of the surgical field of view on a screen displaying the virtual surgical scene in real time, and perform various surgical operations in simulation.

[0050] (6) Surgical field of view:

[0051] The surgical field of view refers to the extent of the target surgical site and surrounding anatomical structures that a surgeon can directly observe with the naked eye or instruments during a surgical procedure. The surgical field of view determines the range and clarity of the surgeon's observation of the target site and surrounding tissues. A good surgical field of view allows the surgeon to observe the anatomical structures of the surgical site more clearly, perform operations more precisely, and avoid damaging important tissues. Appropriate traction, incision location, and correct use of surgical instruments can all help to obtain a larger and clearer surgical field of view.

[0052] In the technical solution of this application, a first-person perspective virtual surgical field of view can be displayed in real time according to the doctor's position and viewpoint in the virtual surgical scene, providing an immersive simulated surgical experience. This allows doctors to adapt to possible surgical field of view conditions in advance, gain a better understanding of important blood vessels and nerve trajectories, and improve the safety of actual surgery.

[0053] The surgical simulation method based on image 3D reconstruction and surgical navigation in the embodiments of this application is described below:

[0054] Please see Figure 1 This is a flowchart illustrating a surgical simulation method based on image 3D reconstruction and surgical navigation in an embodiment of this application.

[0055] S101. Based on the patient's medical imaging data, a three-dimensional model of the lesion site is generated through image segmentation and three-dimensional reconstruction;

[0056] Patient medical imaging data refers to two-dimensional tomographic images of the patient's internal structures acquired through medical imaging equipment such as CT and MRI. Image segmentation refers to the process of separating a target region (such as diseased tissue) from the background in a medical image. Three-dimensional reconstruction refers to the technique of restoring the shape and structure of a three-dimensional object using two-dimensional tomographic images. A three-dimensional model of a lesion site is used to represent a three-dimensional digital representation of the diseased tissue, including information such as its spatial location, shape, and size.

[0057] The surgical simulation system can first acquire the patient's medical imaging data such as CT or MRI; then use image segmentation algorithms to extract the contour of the lesion tissue from these two-dimensional tomographic images; finally, based on the segmentation results, use three-dimensional reconstruction technology to convert the two-dimensional image information into a three-dimensional model, generating a visualized three-dimensional model of the lesion site.

[0058] Understandably, there are many ways to generate a 3D model of the lesion site:

[0059] In some embodiments, a three-dimensional reconstruction method based on multi-layer medical image data can be adopted: first, multi-layer CT or MRI scan images of the patient are acquired, then image segmentation algorithms such as threshold segmentation and region growing are used to extract the contour of the lesion area in each layer of the image, then the three-dimensional shape of the lesion tissue is reconstructed according to the correspondence between adjacent layers of the image, and finally, a visualized three-dimensional model is generated using surface rendering and other techniques.

[0060] Specifically, the process of generating a three-dimensional model of the lesion site begins with acquiring multi-slice CT or MRI scans of the patient. Taking liver tumors as an example, the patient lies flat in the CT scanner, holds their breath for 10-20 seconds, and the scanner rapidly acquires 200-300 cross-sectional DICOM format images covering the entire abdominal area at 2 mm slice thickness and 1 mm intervals. These raw images provide the basic data for subsequent processing.

[0061] After acquiring the images, the next step is image preprocessing. The surgical simulation system can use specialized medical image processing software to read the DICOM file and identify pixel spacing and slice thickness information. Then, Gaussian filtering is applied for noise reduction, replacing the gray value of each pixel with the weighted average of the pixels in its surrounding 3x3 area, effectively reducing random noise. Next, histogram equalization is used to enhance contrast, making the boundaries of the liver and tumor more clearly distinguishable. These preprocessing steps lay the foundation for subsequent accurate segmentation.

[0062] After preprocessing, the crucial image segmentation stage begins. First, the surgical simulation system analyzes the grayscale histogram of the liver region to determine the CT value distribution range of normal liver tissue and tumor tissue, setting an appropriate segmentation threshold. Then, seed points are set at the identified tumor center, and a region growing algorithm is applied to gradually incorporate surrounding pixels that meet the criteria into the tumor region. Finally, morphological closing operations are used to fill small holes and smooth edges, resulting in a more complete tumor contour. This segmentation result provides accurate two-dimensional contour information for subsequent 3D reconstruction.

[0063] With accurate segmentation results, the next step is contour extraction. The Canny edge detection algorithm is applied to the binarized image of each layer to mark the tumor edges. Then, the Moore neighborhood tracking algorithm is used to scan from the top left corner, connecting edge pixels in a clockwise direction to form a complete contour. This step transforms the tumor boundary of each image layer into a set of coordinate points, preparing it for 3D reconstruction.

[0064] After contour extraction, the 3D reconstruction stage begins. Here, the Marching Cubes algorithm can be used to stack the binarized images of all layers into a 3D array, setting an isosurface threshold to generate triangular patches within each voxel unit. This ultimately yields a mesh model composed of numerous small triangles, preliminarily representing the 3D shape of the tumor. To optimize the model, the Laplacian smoothing algorithm can be applied iteratively to make the surface smoother and more natural.

[0065] The reconstructed 3D model requires surface rendering for a visually appealing presentation. This step first calculates the normal vector for each triangular facet, then sets up the Phong lighting model to simulate real-world lighting effects. By adjusting ambient light, diffuse reflection, and specular reflection coefficients, the model achieves a natural three-dimensional effect. Simultaneously, the original CT image is mapped onto the model's surface as a texture, ensuring the 3D model not only has accurate shape but also reflects the grayscale information of the original image.

[0066] Finally, model validation and optimization can be performed. The generated 3D model is projected back onto the original CT image plane, the overlap is calculated, and the model accuracy is evaluated. By adjusting parameters such as the segmentation threshold or the number of smoothing iterations, the model can be repeatedly optimized until a high-quality 3D model is obtained.

[0067] In other embodiments, deep learning models can also be used to construct the three-dimensional model. For example, the patient's DICOM format medical image data can be acquired first, and then a deep learning segmentation network such as U-Net can be used to perform semantic segmentation on the image to identify the lesion area. Then, based on the segmentation mask, a three-dimensional point cloud or mesh model can be generated using methods such as voxelization or mesh reconstruction. Finally, mesh optimization and texture mapping are performed to generate a three-dimensional model of the lesion area.

[0068] Specifically, after obtaining the DICOM data, the surgical simulation system enters the image preprocessing stage. First, it combines all the two-dimensional slices into a three-dimensional volumetric data set, forming a high-dimensional array. Then, the system normalizes this array, mapping CT values ​​to a range of 0 to 1, with air close to 0, bone close to 1, and soft tissue distributed in the middle range. This step not only unifies the data scale but also enhances the contrast between different tissues. Finally, the system may need to interpolate or crop the data to match the input requirements of the deep learning model, preparing it for the next step of semantic segmentation.

[0069] After preprocessing, the surgical simulation system uses a pre-trained U-Net model for semantic segmentation. U-Net's unique architecture enables it to effectively capture multi-scale features of images, making it well-suited for medical image segmentation tasks. The model processes the input 3D image layer by layer, ultimately outputting a probability map where the value of each voxel represents the probability that it belongs to the tumor region. By setting an appropriate threshold, the system converts the probability map into a binary mask, clearly marking the tumor region. This segmentation result provides crucial spatial information for subsequent 3D reconstruction.

[0070] It's important to note that the U-Net model is trained by pre-collecting a large dataset of liver tumor CT or MRI images manually annotated by professional radiologists. This dataset should cover liver tumors of various sizes, shapes, and locations to ensure the model's generalization ability. During training, the system inputs these annotated 3D images into the U-Net model and iteratively adjusts the model parameters using optimization algorithms such as cross-entropy loss and stochastic gradient descent. To improve the model's robustness, data augmentation techniques, such as rotation, scaling, and contrast adjustment, are also employed. During training, the system periodically evaluates the model's performance on a validation set and uses techniques such as early stopping to prevent overfitting.

[0071] With the accurate segmentation mask, the system begins to generate a 3D mesh model. The MarchingCubes algorithm can be used here, which traverses the entire 3D volume, generating triangular patches in each small cube containing the segmentation boundaries. The algorithm examines each 2x2x2 voxel cube, determining how to place triangles based on vertex values, ultimately generating a mesh model composed of numerous small triangles that accurately represents the 3D shape of the tumor.

[0072] The initially generated mesh model is often too complex and requires optimization. The system can apply a mesh simplification algorithm to remove some vertices and edges that have little impact on the overall shape, significantly reducing the model's complexity while preserving key shape features. Then, a Laplacian smoothing algorithm is used for multiple iterations, slightly adjusting vertex positions in each iteration to make the surface smoother and more natural without excessively blurring important details. This optimization process balances model accuracy and computational efficiency.

[0073] To ensure the 3D model is not only accurate in shape but also reflects the information from the original medical image, the system performs texture mapping. Using the original normalized volume data as the texture source, for each vertex on the optimized mesh, its corresponding position in the original 3D image is calculated, and a texture value is calculated using trilinear interpolation. This process assigns a grayscale value to each vertex on the mesh, reflecting the density information at that location in the original CT image, enabling the final model to visually display the internal structure of the tumor and its relationship with surrounding tissues.

[0074] Finally, the surgical simulation system can save the optimized mesh model and texture information in a standard 3D model format, such as OBJ or PLY. This file contains detailed geometric information and texture values ​​for each vertex. The model can be rotated and scaled using the system's interface to observe the tumor's shape, size, location, and its relationship to surrounding tissues from all angles. The texture information provides additional diagnostic value, helping to identify heterogeneity within the tumor or involvement of surrounding tissues.

[0075] It is understandable that other image processing and computer graphics methods can also be used to achieve three-dimensional reconstruction of lesions, and this is not limited here.

[0076] S102. Based on the three-dimensional model of the lesion site, the patient's anatomical data, and the location of important organs, determine multiple optional surgical navigation plans, including different incision locations;

[0077] It is understandable that while generating a three-dimensional model of the lesion site, the surgical simulation system will also generate a three-dimensional anatomical model of the patient as a whole based on the patient's medical imaging data. The generation process is similar to that of generating a three-dimensional model of the lesion site in step S101, and will not be described in detail here.

[0078] Patient anatomical data represents detailed information about the individual patient's organs, blood vessels, nerves, and other anatomical features. The location of vital organs refers to the specific location of key organs relevant to the surgery within the patient's body. Patient anatomical data and the locations of vital organs are determined by the surgical simulation system from a three-dimensional anatomical model generated based on the patient's medical imaging data.

[0079] The surgical simulation system can load a 3D model of the lesion site and spatially register it with the patient's overall 3D anatomical model data. Based on this information, multiple possible incision locations can be generated, each corresponding to a complete surgical navigation plan. These plans consider the path from the incision to the lesion site, evaluating the feasibility, safety, and effectiveness of each path. The surgical simulation system can score each plan, considering factors such as surgical difficulty, impact on surrounding tissues, and surgical time. Ultimately, the surgical simulation system can present several highly-rated optional surgical navigation plans for further evaluation and selection.

[0080] Surgical navigation plans can be generated and evaluated in several ways:

[0081] In some embodiments, the surgical simulation system may employ a rule-based approach: First, based on preset safety zone rules, multiple possible incision locations are determined on the surface of a three-dimensional anatomical model representing the patient as a whole; then, based on a path planning algorithm, one or more paths are designed for each incision location that avoid important organ locations and reach the lesion site in the three-dimensional model; next, based on the patient's anatomical structure data, the length, curvature, and distance to important organ locations of each path are calculated, and a path score is obtained according to a preset scoring standard; finally, multiple paths exceeding a preset scoring threshold are identified as multiple optional surgical navigation schemes.

[0082] Specifically, the surgical simulation system first loads preset safety zone rules, such as avoiding the lower edge of the ribs and the area around the umbilicus in abdominal surgery, or prioritizing specific intercostal spaces in thoracic surgery. These preset safety zone rules are derived from experience accumulated in clinical medical practice, anatomical knowledge, and relevant surgical guidelines. These rules comprehensively consider the structural characteristics of various parts of the human body, the distribution of important organs, and the feasibility of surgical procedures. The surgical simulation system applies these rules to the surface of the three-dimensional anatomical model, marking areas that meet the safety standards and evenly distributing candidate incision points within these areas, while ensuring sufficient operating space around each point.

[0083] After identifying potential incision locations, the next step is to design a path to the lesion site for each incision. This step utilizes an improved A* algorithm to discretize the patient's 3D anatomical model into a voxel mesh and define a cost function that considers distance, tissue type, and safety factors. By fine-tuning the weights of the cost function, multiple different paths can be generated for each incision, providing more options.

[0084] The A* algorithm is a widely used pathfinding and graph traversal algorithm that combines the rigor of Dijkstra's algorithm with the heuristic of best-first search, enabling it to efficiently find the optimal path from the starting point to the target point. The A* algorithm uses an evaluation function f(n) to determine the search order. For each node n, f(n) = g(n) + h(n), where g(n) is the actual cost from the starting point to the current node n; h(n) is the estimated cost from node n to the target (the heuristic function). h(n) is the key part of the algorithm, estimating the cost from the current node to the target. A good heuristic function can significantly improve the algorithm's efficiency. In this application, the A* algorithm can be improved to adapt to three-dimensional space and specific medical constraints: the patient's three-dimensional anatomical model is represented as a three-dimensional voxel mesh, with each voxel representing a possible path point. g(n) can include distance cost, tissue penetration difficulty, etc. h(n) can use straight-line distances in three-dimensional space or more complex estimates considering anatomical structures.

[0085] Subsequently, detailed feature calculations can be performed on each generated path. It measures the path length, calculates the average curvature, and evaluates the minimum and average distances between the path and important anatomical structures.

[0086] After obtaining detailed characteristics of all pathways, the process proceeds to the scheme scoring and ranking stage. Predefined scoring criteria are applied, considering factors such as path length, curvature, and safety distance, to calculate a total score for each scheme. The predefined scoring criteria comprehensively consider key factors such as path length (30%), curvature (25%), safety distance from important structures (30%), tissue penetration difficulty (10%), and surgical field of view (5%), each with its specific calculation method and weight. For example, the path length score is calculated by comparing the actual length with the longest acceptable length, while the safety distance score is based on the minimum distance between the path and key anatomical structures. Then, all schemes are ranked in descending order based on the total score, and the highest-scoring schemes are selected as the final available surgical navigation options.

[0087] In some embodiments, the surgical simulation system may also employ a machine learning-based approach: First, a deep neural network model is trained using a large amount of historical surgical data; then, the three-dimensional model of the lesion site, anatomical structure data, and the location of important organs of the current patient are input into the trained model; next, the model automatically generates multiple surgical navigation plans sorted from high to low by score, including incision location and path planning.

[0088] Specifically, the process begins with large-scale data collection and preprocessing, gathering detailed information from numerous historical surgical cases, including patients' 3D anatomical structures, lesion characteristics, actual surgical plans, and outcomes. This rich data, after standardization and quantization, is used to train a deep neural network. This deep neural network includes: a 3D convolutional neural network (CNN) to process the patient's anatomical data; a point cloud processing network (such as PointNet) to process features of the lesion site; fully connected layers to integrate various features and generate output; and an attention mechanism to focus on important anatomical structures and potential risk areas. The network output can include: a heatmap of the incision location, a sequence of key points along the path, a surgical risk score, and expected surgical time, etc.

[0089] During model training, 80% of the data can be used as the training set, 10% as the validation set, and 10% as the test set. A multi-task loss function is defined: L = λ1L_cutout + λ2L_path + λ3L_risk + λ4L_time, where each sub-loss uses an appropriate metric (e.g., cross-entropy for cutout locations, average distance error for paths). The Adam optimizer is used for training with an initial learning rate of 0.001 and cosine annealing. Dropout (ratio 0.5) and L2 regularization (coefficient 0.0001) are applied to prevent overfitting. The model is trained using the PyTorch framework, evaluated on the validation set every 100 batches, and the model with the lowest validation loss is selected as the final model.

[0090] After model training is complete, the same preprocessing workflow is used to prepare the input data. During model inference, ONNXRuntime can be used for acceleration to generate 10 most probable incision locations and their corresponding paths, serving as multiple surgical navigation options in the final presentation. For each generated path, a Savitzky-Golay filter can be applied for smoothing, and a Fast Marching Method is used to ensure that the path does not cross restricted areas (such as important blood vessels).

[0091] It is understandable that other methods can be used to generate and evaluate surgical navigation plans, such as combining expert systems or hybrid methods, which are not limited here.

[0092] S103. Perform virtual reality simulated surgery based on the first surgical navigation scheme selected by the user. During the simulated surgery, the surgical field of view is displayed according to the user's actual perspective based on the user's operation instructions.

[0093] Among them, operation commands refer to control commands issued by users through specific input devices (such as handles, data gloves, etc.) to manipulate surgical instruments in a virtual environment.

[0094] In some embodiments, the virtual reality simulated surgery can be performed using a VR headset and controllers; in other embodiments, the virtual reality simulated surgery can also be performed using a 3D display and a control handle, without limitation.

[0095] Taking virtual reality simulated surgery using a head-mounted VR headset as an example, the system first loads the relevant data for the user-selected surgical navigation plan, including the patient's 3D anatomical model, the predetermined incision location, and the surgical path. Then, the system initializes the virtual reality environment, converting the loaded data into a format that can be rendered in VR. The user puts on the VR headset, picks up the controller, and enters the virtual operating room. The system sets the user's position and orientation in the virtual environment based on the initial viewpoint of the navigation plan. Subsequently, the user can issue operation commands through the controller, such as moving the viewpoint, selecting and using virtual surgical instruments, etc. The system captures these operation commands in real time, updating the scene and object states in the virtual environment. Simultaneously, the system continuously tracks the user's head movements and dynamically adjusts the rendered viewpoint to ensure that the displayed content always matches the user's actual viewpoint. Throughout the process, the system can also simulate situations that may be encountered in real surgery, such as bleeding and tissue deformation, to enhance the realism of the simulation and the training effect.

[0096] S104. In the event of a failed simulated surgical procedure, display the entire three-dimensional view of the first procedure that caused the failure and reset the simulated surgical progress to before the first procedure was performed.

[0097] In this context, simulated surgical procedures refer to a series of actions performed in a virtual reality environment that mimic the actual surgical process. Surgical failure indicates an unexpected outcome or a potential threat to patient safety during the virtual surgery. The first operation refers to the critical step that led to the surgical failure. A full-process 3D view is used to represent a complete stereoscopic view of the first operation from multiple angles and time points. Resetting the simulated surgical progress means reverting the virtual surgical system's state to a point in time prior to the first operation.

[0098] Specifically, the surgical simulation system first determines whether the surgery has failed based on preset evaluation criteria (such as organ damage, blood loss, and operation time). Once failure is confirmed, the entire surgical process is reviewed to analyze the key operations that led to the failure. Then, the surgical simulation system retrieves the complete 3D record of that operation, including the operator's perspective, hand movements, and instrument trajectory. This information is integrated into a comprehensive 3D view and displayed on the user interface. Simultaneously, the system can mark key error points and provide a comparative view of the correct operation. Finally, the virtual surgery is automatically reverted to the state before the first operation that led to the failure, reloading all relevant data at that point in time, including the patient's physiological state, surgical progress, and instrument positions, providing the user with an opportunity to retry.

[0099] In some embodiments, an expert system knowledge base containing extensive surgical knowledge and experience can be pre-built, defining standard procedures, permissible deviation ranges, and potential risk factors for various operations. When surgery fails, the system starts from the failure outcome and uses a decision tree algorithm to trace the possible causes upwards level by level. At each decision node, key indicators of the current state (such as the degree of organ damage, bleeding volume, vital signs, etc.) are evaluated and compared with thresholds in the knowledge base. Simultaneously, the temporal sequence and cumulative effect of operations are considered to calculate the degree of influence of each operation on these indicators. When the earliest operation that caused the key indicators to exceed the safety threshold is found, it is identified as the first operation leading to the failure.

[0100] In some embodiments, a deep learning model can be used to monitor each step of the user's operation in real time and compare it with standard operating procedures; when a behavior that deviates significantly from the standard operating procedure is detected, a failure alarm is immediately triggered and the key operation sequence that caused the failure is recorded; computer vision technology is used to capture the entire process of the erroneous operation from the perspective of multiple virtual cameras and generate high-precision 3D reconstruction data; natural language processing technology is applied to automatically generate text descriptions and voice explanations of the erroneous operation, which, together with visual replay, provide comprehensive error analysis.

[0101] In some embodiments, an accurate physical model of the surgical environment can be constructed first, including tissue deformation, fluid dynamics, and instrument interaction; changes in all physical parameters are continuously recorded during the simulation to form a complete state history log; when a surgical failure is detected, the physical engine is used to reverse-engineer and accurately locate the initial erroneous operation that caused the failure; based on the state history log, the surgical environment can be accurately rolled back to restore all relevant parameters to the state before the error occurred.

[0102] It is understandable that other methods can be used to display erroneous operations and reset progress, such as combining virtual reality haptic feedback to enhance the perception of erroneous operations, or integrating expert systems to provide personalized improvement suggestions, etc., which are not limited here.

[0103] S105. If the number of simulated surgical operations that perform the first operation fails exceeds the preset failure limit, a recommended second surgical navigation scheme is displayed. The second surgical navigation scheme is a surgical navigation scheme that does not include the first operation among the multiple optional surgical navigation schemes.

[0104] The simulated surgical operation failure count represents the cumulative number of times a specific surgical procedure has been repeatedly performed in the virtual environment, resulting in failure. The preset failure limit refers to the maximum number of failures allowed by the system, used to determine whether further intervention is necessary.

[0105] Specifically, the surgical simulation system continuously monitors the number of times the operator fails in a specific step (i.e., the first operation). After each failure, the system records and updates the failure count. When the number of failures reaches a preset upper limit, the system can trigger a scheme switching mechanism. At this time, the system can filter from multiple pre-prepared optional surgical navigation schemes to select a scheme that does not contain the first operation that caused repeated failures, and display it as a second surgical navigation scheme for the user to choose from. Optionally, the system can also provide a comparison view to show the differences between the old and new schemes, helping the operator understand the reasons and content of the adjustment. After the user selects a new navigation scheme, simulation training can restart based on the new navigation scheme.

[0106] In some embodiments, a rule-based scheme selection method can be adopted: a series of rules are pre-defined to define the applicable conditions and priorities of different surgical navigation schemes; when the number of failures reaches the upper limit, the system automatically triggers the rule engine; the rule engine first excludes all schemes containing failed operations; then, according to preset rules, such as difficulty level, operator experience, etc., the remaining schemes are scored; the scheme with the highest score is selected as the recommended second surgical navigation scheme and displayed to the user.

[0107] In some embodiments, a case-based reasoning approach can be used for solution recommendation: a database containing a large number of historical surgical cases is established, with each case including information such as surgical plan, operator information, and success rate; when a new solution needs to be recommended, the system analyzes the characteristics of the current operator and the reasons for failure; a similarity algorithm is used to find the most similar successful case in the case database; surgical plans that do not contain failed operations are extracted from these similar successful cases; these extracted surgical plans that do not contain failed operations are matched with multiple optional surgical navigation solutions, and the surgical navigation solution with the highest matching degree is displayed to the user as the second surgical navigation solution.

[0108] In this embodiment, a precise 3D model of the lesion site generated from the patient's medical imaging data, combined with the patient's individualized anatomical structure data, enables doctors to intuitively and comprehensively understand the spatial location of the lesion and its relationship with surrounding important tissues, overcoming the difficulty of accurately grasping the spatial relationships of complex cases in traditional methods. The simulated surgery conducted in a virtual reality environment provides doctors with an immersive training experience, while the full-process 3D visual playback and progress reset functions in case of operational failure provide doctors with timely and effective opportunities to correct errors, significantly improving the relevance and interactivity of the training. The mechanism of intelligently recommending alternative surgical navigation schemes in case of repeated failures enhances the flexibility and adaptability of the surgical plan, effectively addressing complex and ever-changing surgical situations. The organic combination of these technical features allows doctors to fully understand the 3D structure of the lesion site before surgery and obtain targeted operational training, improving the effectiveness of surgical simulation.

[0109] In the above embodiments, the surgical simulation system can improve the accuracy of surgical planning and the safety of surgery by integrating 3D reconstruction of patient medical imaging data, multi-plan surgical navigation, and virtual reality simulation technology. However, in practical applications, the selection and coordination of assistant surgeons are also key factors for surgical success, and the surgical simulation system can provide effective assistance in this regard.

[0110] Please see Figure 2 This is another flowchart illustrating the surgical simulation method based on image 3D reconstruction and surgical navigation in the embodiments of this application.

[0111] S201. Based on the patient's medical imaging data, a three-dimensional model of the lesion site is generated through image segmentation and three-dimensional reconstruction.

[0112] S202. Based on the three-dimensional model of the lesion site, the patient's anatomical data, and the location of important organs, determine multiple optional surgical navigation plans, including different incision locations;

[0113] S203. Perform virtual reality simulated surgery based on the first surgical navigation scheme selected by the user. During the simulated surgery, the surgical field of view is displayed according to the user's actual perspective based on the user's operation instructions.

[0114] S204. In the event of a failed simulated surgical procedure, display the entire three-dimensional view of the first procedure that caused the failure and reset the simulated surgical progress to before the first procedure was performed.

[0115] S205. If the number of simulated surgical operations that perform the first operation fails exceeds the preset failure limit, a recommended second surgical navigation scheme is displayed. The second surgical navigation scheme is a surgical navigation scheme that does not include the first operation among the multiple optional surgical navigation schemes.

[0116] Steps S201 to S205 are similar to steps S101 to S105, and will not be described again here.

[0117] S206: Receive user input of a combination of multiple assistant surgeons in the hospital;

[0118] The term "assistant surgeon" refers to the medical staff who assist the lead surgeon in completing the operation, typically including assistant surgeons and scrub nurses. A team of assistant surgeons refers to a group of multiple assistant surgeons.

[0119] Specifically, before the virtual reality simulated surgery begins, the assistant surgeon selection phase can commence. The surgical simulation system can display an assistant surgeon selection interface, listing all available assistant surgeons in the hospital, including their names, titles, and specialties. Users can select multiple assistant surgeons to form a combination using interactive elements such as drop-down menus or checkboxes on the interface. Users can also create multiple different assistant surgeon combinations for subsequent evaluation and comparison. Each time a user completes the selection and confirmation of a combination, the combination information is saved to a temporary database. Users can repeat this process as needed to create multiple different assistant surgeon combinations.

[0120] In some embodiments, the input and management of assistant personnel combinations can be achieved in multiple ways:

[0121] Optionally, a graphical drag-and-drop interface can be provided: the left side of the interface displays the avatars and brief information of all available assistants. The right side features multiple blank "combination areas," each representing an assistant combination. Users can create combinations by dragging avatars from the left to the combination areas on the right. Each combination area has a "save" button; clicking it saves the combination information to the system. Users can modify created combinations at any time, such as adding or deleting members.

[0122] Optionally, a role-based combination generator can be provided: First, common surgical assistant roles are defined, such as "first assistant," "second assistant," and "scaffold nurse." The user selects the desired role combination, such as "2 assistants + 1 scrub nurse." Based on the selected role combination, the system automatically filters suitable candidates from available personnel. The user then selects specific personnel for each role from the candidates. After selection, the system automatically generates a surgical assistant combination and provides a save option.

[0123] It is understandable that other methods can also be used to input and manage the assistant personnel, such as verbal input based on speech recognition, or intelligent recommendation systems combined with artificial intelligence, etc., which are not limited here.

[0124] S207. Based on the historical surgical database, determine the success rate of each assistant surgeon combination participating in each optional surgical navigation scheme;

[0125] Among them, the historical surgical database refers to a storage system containing past surgical records and related information, which is used to analyze and predict the success rate of future surgeries.

[0126] Specifically, after receiving multiple combinations of assistant surgeons input by the user, the system can access the historical surgical database to retrieve all records related to the current surgical type. Then, for each user-input assistant surgeon combination, the system searches the historical data for relevant surgical records involving that combination (or similar combinations). Simultaneously, the system matches these historical records with the currently available surgical navigation options. By analyzing the matched historical data, the system calculates the success rate of each assistant surgeon combination in each available surgical navigation option. This process considers multiple factors, such as surgical complexity, the experience level of the assistant surgeons, and the level of synergy in previous collaborations. Finally, the system generates a matrix displaying the expected success rate of each assistant surgeon combination in each surgical navigation option.

[0127] In some embodiments, surgical success rate analysis based on historical data can be achieved in a variety of ways:

[0128] Optionally, the system can employ a rule-based analysis method: First, define a series of attributes for each assistant surgeon, such as years of experience, area of ​​expertise, and skill rating. Then, set a difficulty coefficient and key skill requirements for each surgical navigation plan. The system filters cases similar to the current surgical type from the historical database. For each combination of assistant surgeons and surgical navigation plans, the system calculates a matching score, considering the fit between the assistant surgeon's attributes and the plan's requirements. Finally, based on the matching score and the proportion of historical successful cases, calculate the expected success rate.

[0129] Optionally, the system can also use a machine learning model for success rate prediction: Collect and preprocess historical surgical data, including information on assistant surgeons, surgical plan details, and surgical outcomes. Use this data to train a machine learning model, such as a random forest or gradient boosting tree. For each new combination of assistant surgeons and surgical navigation plan, its features are fed into the trained model. The model outputs the predicted success rate for that combination under that specific plan. Repeat this process to generate success rate predictions for all combinations and plans.

[0130] It is understandable that other methods can be used to analyze and predict surgical success rates, such as using Bayesian network models to consider the complex dependencies between factors, or combining expert systems with the experience and judgment of medical experts, etc., which are not limited here.

[0131] S208. When displaying the recommended second surgical navigation plan, the surgical navigation plans are sorted according to the patient's planned operation time, the time schedule of each assistant surgeon combination in the current hospital, and the success rate.

[0132] The patient's planned surgery time refers to the specific date and time slot scheduled for the surgery. The current schedule for various auxiliary surgical teams within the hospital refers to the work arrangements for each potential auxiliary surgical team over a period of time.

[0133] When presenting recommended second surgical navigation options to users, the system retrieves the patient's scheduled surgery time and then queries the recent work schedules of various assistant surgeon combinations within the hospital to determine which combinations are available at the scheduled surgery time. Next, it comprehensively evaluates the feasibility and potential effectiveness of each optional surgical navigation option by combining the previously calculated success rates of each combination under different options. During the evaluation process, a weighted algorithm can be used to consider factors such as time matching, personnel availability, and expected success rate. Finally, all feasible surgical navigation options are ranked based on these comprehensive scores, and the results are displayed to the user, along with the reasons for the ranking of each optional surgical navigation option.

[0134] In some embodiments, intelligent sorting of surgical navigation schemes can be achieved in a variety of ways:

[0135] Optionally, a multi-factor weighted scoring method can be used: assign weights to each factor (time matching, personnel availability, and expected success rate), such as 30% for time matching, 30% for personnel availability, and 40% for expected success rate. For time matching, calculate the degree of overlap between the time required for the proposed plan and the patient's planned surgery time; a perfect match receives full marks, while partial overlap receives proportional scores. For personnel availability, check the availability of each team of assistant surgeons during the planned time period; full availability for all personnel receives full marks, while partial availability receives proportional scores. The expected success rate is directly used as the score for this item. The three scores are added together according to their weights to obtain the total score for each plan, and then ranked accordingly.

[0136] Alternatively, a decision tree algorithm can be used for sorting: Construct a decision tree, using time matching degree, personnel availability, and expected success rate as decision nodes. At the first level of the tree, categorize the solutions into "fully matched," "partially matched," and "not matched" based on time matching degree. At the second level, for each category, categorize them into "fully available," "partially available," and "completely unavailable" based on personnel availability. At the third level, use the expected success rate for final sorting. Traverse the decision tree, arranging all solutions in order from best to worst.

[0137] It is understandable that other methods can be used to sort surgical navigation schemes, such as using fuzzy logic to consider the complex relationships between various factors, or applying genetic algorithms to dynamically optimize the sorting results, etc., which are not limited here.

[0138] In some embodiments, during each display of optional surgical navigation schemes, including the initial display of multiple optional surgical navigation schemes, the displayed navigation schemes can be sorted in accordance with steps S206 to S208, which is not limited here.

[0139] S209. When a user selects a surgical navigation scheme for virtual reality simulated surgery, adjust the list of assistant surgeons during the simulated surgery based on the user's modified operations.

[0140] After a user selects a surgical navigation option from the system's recommendations and begins a virtual reality simulated surgery, the system displays a default list of assistant surgeons based on previous settings at the start of the simulation. However, users may wish to adjust the assistant surgeons based on actual circumstances or personal preferences. Specifically, the system provides an interactive interface that allows users to view and modify the current list of assistant surgeons. Users can add new assistant surgeons, remove existing ones, or change specific roles through this interface. Each time a user makes a modification, the system responds immediately, updating the assistant surgeon settings in the virtual environment. This may include updating the appearance of virtual characters, adjusting their positions in the operating room, or even changing their behavior patterns in the simulation. The system can also calculate and display in real-time the potential impact of these changes on the surgical success rate to help users make informed decisions. Users can make adjustments multiple times until they are satisfied with the assistant surgeon configuration before continuing the simulated surgery.

[0141] In some embodiments, the dynamic adjustment of the assistant surgeon during simulated surgery can be achieved in a variety of ways:

[0142] Optionally, the system can provide a graphical drag-and-drop interface: a semi-transparent personnel management panel is displayed on one side of the virtual operating room scene. The panel lists the current assistant surgeons, with a "Remove" button next to each person. At the bottom of the panel is an "Add Personnel" button, which, when clicked, brings up a list of available personnel. Users can drag and drop personnel to adjust their positions within the operating room. After each adjustment, the system updates the personnel configuration and positions in the virtual environment in real time.

[0143] Optionally, the system can also use voice commands and gesture control: users can modify the personnel list using voice commands such as "Add instrument nurse Zhang San". After recognizing the command, the system adds the corresponding role model in the virtual environment. Users can remove a person by pointing to a virtual role and then making a "cross" gesture. The system will confirm each operation with voice feedback, such as "Instrument nurse Zhang San added". Users can adjust the position or responsibilities of virtual roles using specific gesture combinations.

[0144] It is understandable that other methods can be used to dynamically adjust the assistant surgeon, such as using augmented reality technology to add and remove virtual personnel in a real operating room environment, etc., which are not limited here.

[0145] S210. During the simulated surgery, based on the historical operation records of the assistant surgeon in the historical surgical database, the assistant surgeon's corresponding operation is automatically matched and completed.

[0146] Among them, the historical operation records of assistant surgeons refer to a detailed record of each assistant surgeon's specific actions and performance in past surgeries. Self-matching refers to the system automatically selecting and executing operations based on the individual operating habits of the assistant surgeons.

[0147] During virtual reality simulated surgery, the system first identifies the current surgical stage and specific steps. For each virtual assistant surgeon, it queries the historical surgical database to retrieve their historical operation records, analyzes these records, and extracts their typical behavioral patterns and personal habits at similar surgical stages. This may include their preferred positioning, instrument delivery methods, reaction speed, etc. Next, based on the extracted personal operating habits, the system generates corresponding operation sequences for each assistant surgeon in the virtual environment. These operations may not always perfectly align with the lead surgeon but realistically reflect each assistant surgeon's individual characteristics. Finally, the system executes these self-matching auxiliary operations in the virtual reality environment, making each virtual assistant surgeon's behavior appear consistent with their personal style. This method allows the lead surgeon to experience the challenge of collaborating with assistant surgeons of different characteristics during training, thereby improving adaptability and coordination.

[0148] In some embodiments, self-matching of assistant surgeon operations based on personal habits can be achieved in a variety of ways:

[0149] Optionally, the system can use a statistically based personal habit model: data mining is performed on the historical operation records of each assistant surgeon to extract frequent operation patterns and time distributions. An operation probability distribution model is established for each type and stage of surgery for that assistant surgeon. During simulated surgery, the system identifies the current surgical stage and randomly samples from the corresponding probability distribution to determine the next operation. The system considers the time interval of the operation to simulate the assistant surgeon's reaction speed and rhythm. The sampled operation is executed, and the results are recorded for subsequent model updates and optimizations.

[0150] Optionally, the system can also employ a personalized reinforcement learning model: a unique reinforcement learning agent is trained for each assistant surgeon, with the initial policy based on their historical operational data. A state space (including surgical stage, instrument position, etc.) and an action space (various possible auxiliary operations) are defined. During the simulation, the agent selects an action based on the current state, but introduces a degree of randomness to simulate human uncertainty. The system executes the selected action and rewards the agent based on whether it aligns with the assistant surgeon's historical habits. The agent continuously learns and adapts, gradually forming behavioral patterns that better suit the individual surgeon's characteristics.

[0151] It is understandable that other methods can also be used to achieve self-matching of assistant operations based on personal habits, such as using generative adversarial networks (GANs) to generate operation sequences that conform to the style of a specific assistant, or combining fuzzy logic systems to simulate the uncertainty of human decision-making, etc., which are not limited here.

[0152] In some embodiments, steps S209-S210 may not be executed; in some embodiments, steps S209-S210 may be executed whenever the user selects a surgical navigation scheme to perform virtual reality simulated surgery, which is not limited here.

[0153] In this embodiment, by employing historical data-based assessment of assistant surgeon combinations and personalized operation simulation technology, users can select the best assistant team and realistically simulate the individual operating habits of each assistant surgeon in a virtual environment. This improves the realism and effectiveness of surgical simulation training, thereby comprehensively enhancing the quality of surgical rehearsals, providing more comprehensive, accurate, and personalized preoperative preparation, and ultimately reducing surgical risks.

[0154] The following describes an exemplary surgical simulation system 100 provided in an embodiment of this application. Figure 3 This is an exemplary hardware structure diagram of the surgical simulation system 100 provided in this application embodiment.

[0155] In some embodiments, the surgical simulation system 100 includes a computer device. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores data. The network interface of the computer device is used to communicate with other external terminals or servers via a network connection. In some embodiments, the network interface can be a wired network interface; in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, it implements the methods described in the embodiments of this application.

[0156] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0157] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0158] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0159] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0160] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A surgical simulation method based on image 3D reconstruction and surgical navigation, characterized in that, include: S201. Based on the patient's medical imaging data, a three-dimensional model of the lesion site is generated through image segmentation and three-dimensional reconstruction. S202. Based on the three-dimensional model of the lesion site, the patient's anatomical data, and the location of important organs, determine multiple optional surgical navigation schemes, including different incision locations; S203. Perform virtual reality simulated surgery based on the first surgical navigation scheme selected by the user. During the simulated surgery, the surgical field of view is displayed according to the user's actual perspective based on the user's operation instructions. S204. In the event of a failed simulated surgical procedure, display the entire three-dimensional view of the first procedure that caused the failure, and reset the simulated surgical progress to before the first procedure was performed. S205. If the number of simulated surgical operations that perform the first operation fails exceeds the preset failure limit, a recommended second surgical navigation scheme is displayed based on the ranking of surgical navigation schemes. The second surgical navigation scheme is the surgical navigation scheme that does not include the first operation among the multiple optional surgical navigation schemes. The method further includes: SA1, after the user selects the next surgical navigation scheme from the recommended second surgical navigation scheme, displaying a comparison view, the comparison view including the differences between the next surgical navigation scheme and the first surgical navigation scheme; SA2, resetting the virtual reality simulated surgery based on the next surgical navigation scheme; The method further includes: S206: Receive user input of a combination of multiple assistant surgeons in the hospital; S207. Based on the historical surgical database, determine the success rate of each assistant surgeon combination participating in each optional surgical navigation scheme; S208. When displaying the recommended second surgical navigation plan, the surgical navigation plans are sorted according to the patient's planned surgery time, the time schedule of each assistant surgeon combination in the current hospital, and the success rate.

2. The method according to claim 1, characterized in that, Based on the three-dimensional model of the lesion site, the patient's anatomical data, and the location of vital organs, multiple optional surgical navigation schemes, including different incision locations, are determined, specifically including: Based on the preset safe zone rules, multiple possible incision locations are determined on the surface of the three-dimensional anatomical model representing the patient as a whole; Based on path planning algorithms, one or more paths are designed for each incision location to reach the lesion site in the 3D model of the lesion site, avoiding the location of important organs; Based on the patient's anatomical data, the length, curvature, and distance to vital organs of each path are calculated, and a path score is obtained according to a preset scoring standard. Multiple paths exceeding the preset scoring threshold are identified as the multiple optional surgical navigation schemes.

3. The method according to claim 1, characterized in that, The aforementioned display of the surgical field of view according to the user's actual perspective specifically includes: Real-time acquisition of user head position, rotation angle, or eye movement data to obtain perspective information; wherein, the perspective information refers to the user's observation position and observation direction in the virtual environment, the observation position is determined by the head position, and the observation direction is determined by the rotation angle or eye movement data; Based on the perspective information, the virtual camera parameters are updated, including the camera's position and orientation in the virtual environment; wherein, a virtual camera refers to the observation point that simulates a real camera in a three-dimensional virtual environment; The surgical scene was re-rendered using the updated virtual camera parameters and then displayed.

4. The method according to claim 1, characterized in that, In the event of a simulated surgical procedure failure, the system displays a three-dimensional view of the entire process of the first procedure that led to the failure, specifically including: If a surgical procedure is determined to have failed based on preset evaluation criteria, the entire surgical process is reviewed to identify the first procedure that led to the failure. Access the complete 3D record of the first operation, including the operator's perspective, hand movements, and instrument trajectory; Based on the complete 3D record of the first operation, a full 3D view of the first operation is generated and displayed on the user interface.

5. The method according to any one of claims 1 to 4, characterized in that, The scheme also includes: When a user selects a surgical navigation scheme for virtual reality simulated surgery, the list of assistant surgeons during the simulated surgery is adjusted based on the user's modified operations. During the simulated surgery, the system automatically matches and completes the corresponding operations of the assistant surgeon based on the historical operation records of the assistant surgeon in the historical surgical database.

6. A surgical simulation system, characterized in that, The surgical simulation system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the surgical simulation system to perform the method as described in any one of claims 1-5.

7. A computer program product containing instructions, characterized in that, When the computer program product is run on the surgical simulation system, the surgical simulation system performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on the surgical simulation system, the surgical simulation system performs the method as described in any one of claims 1-5.

Images