Operation simulation method and system based on image three-dimensional reconstruction and operation navigation
Through three-dimensional reconstruction and multi-scheme surgical navigation technology based on medical image data, combined with virtual reality environment and intelligent feedback mechanism, the problem of lack of precise spatial information and diversified solutions in surgical simulation is solved, and the accuracy and learning efficiency of surgical simulation are improved .
Patent Information
- Application Number
- CN202411159565.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-08-22
AI Technical Summary
The existing surgical simulation methods lack precise spatial information and diverse surgical plans, resulting in low surgical success rate and learning efficiency, especially in complex cases.
The lesion site model is generated through three-dimensional reconstruction technology based on medical image data, combining multi-scheme surgical navigation algorithm and intelligent feedback mechanism, providing multiple optional surgical navigation solutions, and dynamic viewing angle display and failed operation playback in a virtual reality environment. Recommended alternatives.
It significantly improves the accuracy, flexibility and teaching effect of surgical simulation, enhances doctors' learning efficiency and surgical skills, reduces the workload of preoperative planning, and improves the scientificity and feasibility of surgical plans.
Smart Images

Figure CN119014978A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of biomedical engineering and virtual reality processing software, and in particular to a surgical simulation method and system based on three-dimensional image reconstruction and surgical navigation. Background Art
[0002] With the continuous advancement of medical technology, surgical simulation has been widely used in the medical field as an important means of preoperative preparation and training. Surgical simulation can help doctors better understand the patient's condition, develop appropriate surgical plans, and improve the success rate and safety of surgery.
[0003] A common surgical simulation method is to plan surgery based on the patient's medical imaging data. Doctors analyze the patient's CT or MRI images and combine their own experience to determine the location and extent of the lesion and develop a preliminary surgical plan.
[0004] In actual operation, doctors need to construct a three-dimensional image of the patient's internal structure in their minds based on the images, which requires doctors to have high spatial imagination and experience. Especially when dealing with complex cases, if doctors cannot accurately grasp the precise spatial relationship between the lesion and the surrounding important tissues, the risk of misjudgment during the operation will increase. For example, in neurosurgery, errors at the millimeter level can lead to serious consequences. Summary of the invention
[0005] The present application provides a surgical simulation method and system based on three-dimensional image reconstruction and surgical navigation, which is used to enable doctors to fully understand the three-dimensional structure of the lesion site before surgery and obtain targeted operation training, thereby improving the effectiveness of surgical simulation.
[0006] In a first aspect, the present application provides a surgical simulation method based on image three-dimensional reconstruction and surgical navigation, comprising: generating a three-dimensional model of a lesion site through image segmentation and three-dimensional reconstruction based on patient medical image data; determining a plurality of optional surgical navigation schemes including different incision positions according to the three-dimensional model of the lesion site, the patient's anatomical structure data and the position of important organs; performing virtual reality simulated surgery according to a first surgical navigation scheme selected by a user, and during the simulated surgery, displaying the surgical field of view according to the user's actual perspective based on the user's operating instructions; in the event of a failure of the simulated surgical operation, displaying the three-dimensional field of view of the entire process of the first operation that caused the failure of the operation, and resetting the progress of the simulated surgery to before the first operation was performed; in the event that the number of simulated surgical operation failures of the first operation exceeds a preset upper limit of failures, displaying a recommended second surgical navigation scheme, which is a surgical navigation scheme that does not include the first operation among the multiple optional surgical navigation schemes.
[0007] By adopting the above technical solutions, the conversion from patient medical imaging data to three-dimensional models is realized, providing an accurate spatial information basis for surgical simulation. Based on this, the system can generate multiple optional surgical navigation schemes, fully considering the feasibility of different incision positions. During the virtual reality simulation surgery, the system dynamically displays according to the user's actual perspective, greatly improving the realism and immersion of the simulation. When the operation fails, the system can not only replay the entire process of the failed operation, but also intelligently recommend alternatives. This feedback and optimization mechanism significantly improves the doctor's learning efficiency and surgical skills. Overall, this method greatly improves the accuracy, flexibility and effectiveness of surgical simulation by combining image reconstruction, multi-scheme navigation and intelligent feedback.
[0008] In combination with some embodiments of the first aspect, in some embodiments, based on the three-dimensional model of the lesion site, the patient's anatomical structure data and the location of important organs, multiple optional surgical navigation schemes including different incision positions are determined, specifically including: according to a preset safety area rule, multiple possible incision positions are determined on the surface of the three-dimensional anatomical model representing the patient as a whole; based on a path planning algorithm, one or more paths are designed for each incision position that avoid the location of important organs and reach the lesion site in the three-dimensional model of the lesion site; based on the patient's anatomical structure data, the length, curvature and distance of each path from the location of important organs are calculated, and a path score is obtained according to a preset scoring standard; and multiple paths that exceed a preset scoring threshold are determined as the multiple optional surgical navigation schemes.
[0009] By adopting the above technical solutions, the system can accurately locate multiple possible incision locations on the surface of the patient's three-dimensional anatomical model based on the preset safety zone rules. The application of the path planning algorithm ensures that each incision location has one or more safe paths that avoid important organs to reach the lesion site. By calculating the length, curvature and distance from important organs of the path, and evaluating according to preset scoring criteria, the system can screen out the optimal surgical navigation plan. This method not only takes into account the safety of the operation, but also optimizes the efficiency and accuracy of the operation. By automatically generating and evaluating multiple surgical plans, this technology greatly reduces the workload of doctors in the preoperative planning stage, while improving the scientificity and feasibility of surgical plans.
[0010] In combination with some embodiments of the first aspect, in some embodiments, the surgical field of view is displayed according to the user's actual perspective, specifically including: real-time acquisition of the user's head position, rotation angle or eye movement data to obtain perspective information; wherein the perspective information refers to the user's observation position and observation direction in the virtual environment, the observation position is determined by the head position, and the observation direction is determined by the rotation angle or eye movement data; according to the perspective information, the virtual camera parameters are updated, and the virtual camera parameters include the position and orientation of the camera in the virtual environment; wherein the virtual camera refers to an observation point that simulates a real camera in a three-dimensional virtual environment; and the surgical scene is re-rendered using the updated virtual camera parameters and pushed for display.
[0011] By adopting the above technical solutions, the system can capture the user's head position, rotation angle or eye movement data in real time, so as to accurately obtain the user's observation position and direction in the virtual environment. These data are used to update the virtual camera parameters to ensure that the perspective in the virtual environment is synchronized with the user's actual perspective. By rendering and pushing updated surgical scenes in real time, the system creates a highly realistic visual experience. This dynamic perspective adjustment technology not only improves the realism of simulated surgery, but also allows users to observe and operate the virtual surgical environment in the most natural way. This greatly enhances the effect of simulation training, enabling doctors to better adapt to the real surgical environment and improve their surgical skills.
[0012] In combination with some embodiments of the first aspect, in some embodiments, in the case of a simulated surgical operation failure, a three-dimensional field of view of the entire process of the first operation that caused the failure of the operation is displayed, specifically including: when the surgical operation failure is determined based on preset evaluation criteria, the entire surgical process is traced back to determine the first operation that caused the failure of the operation; the complete three-dimensional record of the first operation is called, including the operator's perspective, hand movements, and instrument trajectories; based on the complete three-dimensional record of the first operation, a three-dimensional field of view of the entire process of the first operation is generated, and displayed on the user interface.
[0013] By adopting the above technical solutions, the system can quickly locate the key operations that caused the failure when the surgical operation fails. By retracing the entire surgical process and calling the complete three-dimensional record, the system can accurately reproduce the entire process of the failed operation, including the operator's perspective, hand movements, and instrument trajectories. This comprehensive three-dimensional field of view playback not only allows users to intuitively understand the cause of failure, but also provides rich visual information for in-depth analysis and improvement. In this way, the system greatly improves learning efficiency, allowing doctors to learn from failures and quickly improve their techniques. This immediate and detailed feedback mechanism greatly enhances the educational value and practicality of surgical simulation.
[0014] In combination with some embodiments of the first aspect, in some embodiments, the method further includes: after the user selects a next surgical navigation plan from the recommended second surgical navigation plan, displaying a comparison view, the comparison view including the differences between the next surgical navigation plan and the first surgical navigation plan; and resetting the virtual reality simulated surgery based on the next surgical navigation plan.
[0015] By adopting the above technical solutions, the system provides an intuitive comparison view when recommending new surgical navigation plans, clearly showing the differences between the old and new plans. This visual comparison enables doctors to quickly understand the pros and cons of different plans, providing strong support for decision-making. At the same time, the system is able to reset the virtual reality simulation surgery based on the newly selected plan, allowing doctors to try new surgical paths immediately. This flexible switching and comparison mechanism greatly improves the efficiency and accuracy of surgical plan selection, allowing doctors to more comprehensively evaluate different surgical strategies and choose the best plan. This not only optimizes the preoperative planning process, but also improves the quality of the final surgical plan.
[0016] In combination with some embodiments of the first aspect, in some embodiments, the scheme also includes: when a user selects a surgical navigation scheme to perform virtual reality simulated surgery, adjusting the list of assistant surgeons during the simulated surgery according to the user's modification operation; during the simulated surgery, self-matching is performed to complete the corresponding operations of the assistant surgeon based on the historical operation records of the assistant surgeon in the historical surgery database.
[0017] By adopting the above technical solutions, the system allows users to flexibly adjust the list of assistant surgeons in simulated surgery, increasing the authenticity and adaptability of the simulation. More importantly, the system can automatically match and complete the corresponding operations of assistant surgeons based on the records in the historical surgery database. This intelligent auxiliary function not only improves the integrity and realism of the simulated surgery, but also enables the surgeon to better rehearse and adapt to the cooperation mode of different auxiliary teams. By simulating a real team collaboration environment, this technology significantly enhances the practical value of surgical simulation and helps to improve the coordination ability and overall efficiency of the entire surgical team.
[0018] In combination with some embodiments of the first aspect, in some embodiments, the method also includes: receiving multiple auxiliary surgeon combinations in the hospital input by a user; determining, based on the historical surgical database, the success rate of each auxiliary surgeon combination participating in each optional surgical navigation plan; and when displaying the recommended second surgical navigation plan, sorting the surgical navigation plans according to the patient's planned surgical time, the time schedule of each auxiliary surgeon combination in the current hospital, and the success rate.
[0019] By adopting the above technical solutions, the system can comprehensively consider multiple key factors, including the historical success rate of different auxiliary knife personnel combinations, the patient's planned operation time, and the hospital's current staff scheduling. With these data, the system can intelligently sort the recommended surgical navigation plans. This multi-dimensional evaluation and sorting mechanism not only considers the technical feasibility of the operation itself, but also fully considers the actual situation of hospital resources. Such intelligent recommendations greatly improve the efficiency and accuracy of surgical planning, help hospitals optimize resource allocation, and improve the overall success rate of operations. At the same time, it also provides doctors with more comprehensive decision-making support, which helps to select the best surgical plan and team combination.
[0020] In a second aspect, the present application provides a surgical simulation system, which comprises: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code comprises computer instructions, and the one or more processors call the computer instructions to enable the surgical simulation system to perform the method described in the first aspect and any possible implementation method of the first aspect.
[0021] In a third aspect, the present application provides a computer program product comprising instructions, which, when executed on a surgical simulation system, enables the surgical simulation system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0022] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a surgical simulation system, causes the surgical simulation system to execute the method described in the first aspect and any possible implementation of the first aspect.
[0023] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Due to the use of three-dimensional reconstruction technology based on medical imaging data, multi-scheme surgical navigation generation algorithm and intelligent failure feedback and optimization mechanism, the technical problems of lack of accurate spatial information, single surgical plan and low learning efficiency in surgical simulation in the existing technology are effectively solved, thereby achieving high-precision, multi-scheme and intelligent feedback surgical simulation effects, significantly improving the accuracy, flexibility and teaching effect of surgical simulation.
[0024] 2. Due to the use of automatic recognition technology for surgical failure based on preset evaluation standards, a full-process three-dimensional surgical backtracking mechanism, and accurate reproduction and visualization technology of failed operations, the technical problems of untimely surgical simulation feedback, unintuitive analysis of failure causes, and low learning efficiency in the existing technology are effectively solved, thereby achieving instant, comprehensive, and intuitive surgical failure analysis and learning effects, and significantly improving the educational value and practicality of surgical simulation.
[0025] 3. Due to the use of adjustable auxiliary knife personnel configuration function and intelligent auxiliary operation matching technology based on historical surgical database, the technical problems of lack of team collaboration environment and insufficient realism of auxiliary operation in existing technologies in surgical simulation are effectively solved, thereby achieving a highly simulated surgical team collaboration simulation effect, and significantly enhancing the practical value of surgical simulation and the effect of team coordination ability training. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a flowchart of a surgical simulation method based on three-dimensional image reconstruction and surgical navigation in an embodiment of the present application; Figure 2 is another flowchart of a surgical simulation method based on three-dimensional image reconstruction and surgical navigation in an embodiment of the present application; Figure 3 It is a schematic diagram of an exemplary hardware structure of a surgical simulation system in an embodiment of the present application. DETAILED DESCRIPTION
[0027] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to and includes any or all possible combinations of one or more listed items.
[0028] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.
[0029] To facilitate understanding, the relevant terms and concepts involved in the embodiments of the present application are first introduced below.
[0030] (1) Image segmentation: Image segmentation is the process of decomposing a digital image into multiple regions with similarities, so that each region corresponds to an object or subject in the image. It is a fundamental technique in image analysis and understanding.
[0031] The goal of image segmentation is to simplify or change the representation of an image to make it more meaningful and easier to analyze. It decomposes the image into a set of disjoint regions, where each region usually corresponds to an object or object in the image. Pixels in an image are grouped together according to certain criteria to form a segmented region.
[0032] In the technical solution of the present application, it is necessary to extract the diseased tissue area based on the patient's medical image data such as CT or MRI scan images through image segmentation technology, and use the segmentation results to build a three-dimensional model of the lesion site. For example, the scanned image can be processed using algorithms such as threshold segmentation and edge detection to identify the contours of diseased tissues such as tumors, and then segment them from the background normal tissue to generate a three-dimensional model of the lesion site. This provides important basic data for subsequent surgical simulation and navigation.
[0033] (2) 3D reconstruction: Three-dimensional reconstruction refers to the process of restoring the shape and internal structure of a three-dimensional object from a two-dimensional tomographic image using image processing technology.
[0034] 3D reconstruction technology analyzes the structural information in 2D images and establishes the relationship between images to restore the 3D scene. In the medical field, it can reconstruct the 3D digital model of human tissues and organs through 2D tomography images such as X-rays, CT, and MRI. 3D reconstruction provides doctors with an interactive 3D view, which helps with lesion location, surgical planning, etc.
[0035] In the technical solution of the present application, it is necessary to generate a three-dimensional digital model of the lesion (such as a tumor) based on the patient's CT or MRI scan image using three-dimensional reconstruction technology. First, the contour of the lesion tissue is extracted through image segmentation; then, according to the image shooting parameters, segmentation results, and the positional relationship of adjacent slices, the corresponding points in each layer of the image are determined to restore the three-dimensional shape; finally, the generated three-dimensional digital model is used to establish a virtual scene of the lesion to provide data support for subsequent surgical simulation and navigation. This method can accurately display the three-dimensional position, size, and shape of the lesion, which is conducive to improving the effect of surgical plan formulation.
[0036] (3) Incision location: The incision location refers to the location on the outside of the human body where the incision is made to enter the human body for operation. Choosing the correct incision location is very important for surgery, as it is related to whether the target lesion can be easily reached and whether important organs and blood vessels can be damaged. The selection of the incision location needs to consider multiple factors such as the location of the lesion and the surgical approach. A good incision location can create a good field of view for surgery, making the operation smooth and safe.
[0037] In the technical solution of the present application, it is necessary to determine multiple optional incision positions as different surgical navigation schemes based on the three-dimensional model of the lesion site, the patient's anatomical structure data, etc. For example, for cranial brain tumors, parietal incisions or occipital incisions can be selected; for abdominal tumors, midline incisions or perforation incisions can be selected. The best incision can be selected based on the location of the lesion and the requirements of the operation. In this way, the incision can be planned in advance, and targeted surgical simulation training can be carried out to improve the surgical effect.
[0038] (4) Surgical navigation plan: A surgical navigation plan refers to a navigation path and surgical procedure planning plan for a specific operation. The surgical navigation plan plans each step of the operation in advance, including the incision location, incision path, specific operation sequence, etc., based on the location and nature of the lesion and the individual anatomical characteristics of the patient. A good surgical navigation plan can avoid important blood vessels and nerves to the greatest extent possible, and directly and safely reach the lesion site for treatment.
[0039] In the technical solution of this application, it is necessary to determine multiple optional surgical navigation schemes based on the 3D model of the lesion and the anatomical structure data, and each scheme corresponds to a different incision position. For example, kidney cancer resection surgery can have laparoscopic incision schemes and open incision schemes, each of which includes the precise incision position, the path to the tumor, precautions, etc. Doctors can choose the best plan based on the simulation results, conduct targeted training, master the situations that may be encountered in various surgeries, and improve the success rate of the surgery.
[0040] (5) Virtual reality simulation surgery: Virtual reality surgery simulation uses virtual reality technology to simulate the surgical process in a virtual environment. Doctors can perform simulated surgery through the surgical simulation system, and the system can provide instant feedback based on different situations. This allows doctors to familiarize themselves with the surgical process in advance, evaluate surgical plans, and improve surgical skills.
[0041] In the technical solution of the present application, a virtual surgical scene including a three-dimensional model of the lesion can be constructed, and a simulated surgery can be performed according to the selected surgical navigation scheme. Doctors can wear VR equipment or directly see the surgical field of view from a first-person perspective in real time through the screen displaying the virtual surgical scene, and simulate various surgical operations.
[0042] (6) Surgical field of view: The surgical field of view refers to the range of the target surgical site and surrounding anatomical structures that the doctor can directly observe with the naked eye or instruments during the operation. The surgical field of view determines the scope and clarity of the doctor's observation of the target site and surrounding tissues. A good surgical field of view allows the doctor to observe the anatomical structure of the surgical site more clearly, perform operations more accurately, and avoid damaging important tissues. Appropriate traction, incision location, and correct use of surgical instruments can help obtain a larger and clearer surgical field of view.
[0043] In the technical solution of this application, the virtual surgical field of view from the first-person perspective can be displayed in real time according to the doctor's position and perspective in the virtual surgical scene, providing an immersive simulated surgical experience. This allows doctors to adapt to possible surgical field conditions in advance and have a better understanding of important blood vessels and nerve trajectories, thereby improving the safety of actual surgery.
[0044] The following describes the surgical simulation method based on three-dimensional image reconstruction and surgical navigation in the embodiment of the present application: See also Figure 1 , which is a flow chart of a surgical simulation method based on three-dimensional image reconstruction and surgical navigation in an embodiment of the present application.
[0045] S101, generating a three-dimensional model of the lesion site through image segmentation and three-dimensional reconstruction based on the patient's medical imaging data; Among them, patient medical imaging data refers to the two-dimensional tomographic images of the patient's internal structure obtained by medical imaging equipment such as CT and MRI. Image segmentation refers to the process of separating the target area (such as lesion tissue) in the medical image from the background. Three-dimensional reconstruction refers to the technology of restoring the shape and structure of three-dimensional objects using two-dimensional tomographic images. The three-dimensional model of the lesion site is used to represent the three-dimensional digital representation of the lesion tissue, including its spatial position, shape and size.
[0046] The surgical simulation system can first obtain the patient's medical imaging data such as CT or MRI; then use the image segmentation algorithm to extract the contour of the diseased tissue from these two-dimensional tomographic images; finally, based on the segmentation results, use three-dimensional reconstruction technology to convert the two-dimensional image information into a three-dimensional model to generate a visual three-dimensional model of the lesion site.
[0047] It is understandable that there are many specific ways to generate a three-dimensional model of the lesion: In some embodiments, a three-dimensional reconstruction method based on multi-layer medical imaging data may be used: first, the patient's multi-layer CT or MRI scan images are acquired, and then image segmentation algorithms such as threshold segmentation and region growing are used to extract the contour of the lesion area in each layer of the image. Then, based on the correspondence between adjacent layer images, the three-dimensional shape of the lesion tissue is reconstructed. Finally, surface rendering and other techniques are used to generate a visual three-dimensional model.
[0048] Specifically, the process of generating a 3D model of the lesion begins with the acquisition of multi-slice CT or MRI scan images of the patient. Taking liver tumors as an example, the patient lies flat in the CT scanner and holds his breath for 10-20 seconds, allowing the scanner to quickly acquire 200-300 cross-sectional DICOM format images covering the entire abdominal area with a 2 mm slice thickness and 1 mm interval. These raw images provide basic data for subsequent processing.
[0049] After obtaining the image, the next step is to preprocess the image. The surgical simulation system can use professional medical image processing software to read DICOM files and identify pixel spacing and layer thickness information. Then, Gaussian filtering is applied for denoising, replacing the grayscale value of each pixel with the weighted average of the pixels in the surrounding 3x3 area, effectively reducing random noise. Next, the contrast is enhanced by the histogram equalization method to make the boundaries of the liver and tumor more clearly identifiable. These preprocessing steps lay the foundation for subsequent accurate segmentation.
[0050] After preprocessing, the critical image segmentation stage begins. First, the surgical simulation system determines the CT value distribution range of normal liver tissue and tumor tissue by analyzing the grayscale histogram of the liver area, and sets an appropriate segmentation threshold. Then, a seed point is set at the center of the identified tumor, and a region growing algorithm is applied to gradually incorporate surrounding pixels that meet the conditions into the tumor area. Finally, a morphological closing operation is used to fill small holes and smooth the edges to obtain a more complete tumor contour. This segmentation result provides accurate two-dimensional contour information for subsequent three-dimensional reconstruction.
[0051] With accurate segmentation results, the next step is contour extraction. The Canny edge detection algorithm is applied to each layer of the binary image to mark the edge of the tumor. Then the Moore neighborhood tracking algorithm is used to scan from the upper left corner and connect the edge pixels in a clockwise direction to form a complete contour. This step converts the tumor boundary of each layer of the image into a series of coordinate point sets, preparing for 3D reconstruction.
[0052] After the contour extraction is completed, the 3D reconstruction stage begins. Here, the Marching Cubes algorithm can be used to stack the binary images of all layers into a 3D array, set the isosurface threshold, and generate triangular patches in each voxel unit. Finally, a mesh model composed of a large number of small triangles is obtained, which preliminarily shows the 3D shape of the tumor. In order to optimize the model, the Laplace smoothing algorithm can also be applied for multiple iterations to make the surface smoother and more natural.
[0053] The reconstructed 3D model needs to be rendered to be intuitive. This step first calculates the normal vector of each triangular face, and then sets the Phong lighting model to simulate the lighting effect of the real world. By adjusting the ambient light, diffuse reflection and specular reflection coefficients, the model presents a natural three-dimensional sense. At the same time, the original CT image is mapped to the model surface as a texture, so that the 3D model is not only accurate in shape, but also reflects the grayscale information of the original image.
[0054] Finally, the model can be verified and optimized. The generated 3D model is projected back to the original CT image plane, the overlap is calculated, and the model accuracy is evaluated. By adjusting parameters such as the segmentation threshold or the number of smoothing iterations, the optimization can be repeated until a high-quality 3D model is obtained.
[0055] In other embodiments, a deep learning model may also be used to complete the construction of a three-dimensional model. For example, the patient's DICOM format medical imaging data may be obtained first, and then a deep learning segmentation network such as U-Net may be used to perform semantic segmentation on the image and identify the lesion area. Based on the segmentation mask, a three-dimensional point cloud or mesh model may be generated by using voxelization or mesh reconstruction methods. Finally, mesh optimization and texture mapping may be performed to generate a three-dimensional model of the lesion site.
[0056] Specifically, after obtaining the DICOM data, the surgical simulation system enters the image preprocessing stage. First, it combines all two-dimensional slices into a three-dimensional volume data to form a high-dimensional array. The system then standardizes this array and maps the CT values between 0 and 1, so that the air is close to 0, the bone is close to 1, and the soft tissue is distributed in the middle range. This step not only unifies the data scale, but also enhances the contrast between different tissues. Finally, the system may need to interpolate or crop the data to match the input requirements of the deep learning model and prepare for the next step of semantic segmentation.
[0057] After preprocessing, the surgical simulation system uses a pre-trained U-Net model for semantic segmentation. The unique architecture of U-Net enables it to effectively capture the multi-scale features of the image, making it very suitable for medical image segmentation tasks. The model processes the input 3D image layer by layer and ultimately outputs a probability map, where the value of each voxel represents the probability that it belongs to the tumor area. By setting a suitable threshold, the system converts the probability map into a binary mask that clearly marks the tumor area. This segmentation result provides key spatial information for subsequent 3D reconstruction.
[0058] It should be noted that the training of the U-Net model can be carried out by pre-collecting a large number of liver tumor CT or MRI image datasets with manual annotations by professional radiologists. These datasets should cover liver tumors of various sizes, shapes and locations to ensure the generalization ability of the model. During the training process, the system inputs these annotated three-dimensional images into the U-Net model, and uses optimization algorithms such as cross entropy loss function and stochastic gradient descent to iteratively adjust the model parameters. In order to improve the robustness of the model, data enhancement techniques such as rotation, scaling and contrast adjustment are also used. During the training process, the system will regularly evaluate the model performance on the validation set and use techniques such as early stopping to prevent overfitting.
[0059] With an accurate segmentation mask, the system begins to generate a 3D mesh model. The MarchingCubes algorithm can be used here, which traverses the entire 3D volume and generates triangular patches in each small cube containing the segmentation boundary. The algorithm checks each 2x2x2 voxel cube and determines how to place triangles based on the vertex values, ultimately generating a mesh model composed of a large number of small triangles that accurately represents the 3D shape of the tumor.
[0060] The initially generated mesh model is often too complex and needs to be optimized. The system can apply a mesh simplification algorithm to delete some vertices and edges that have little effect on the overall shape, significantly reducing the complexity of the model while maintaining key shape features. Then use the Laplace smoothing algorithm for multiple iterations, each iteration slightly adjusting the vertex position to make the surface smoother and more natural without overly blurring important details. This optimization process balances the accuracy of the model with computational efficiency.
[0061] In order to make the 3D model not only accurate in shape but also reflect the information of the original medical image, the system will perform texture mapping. Using the original standardized volume data as the texture source, for each vertex on the optimized mesh, calculate its corresponding position in the original 3D image, and calculate the texture value through trilinear interpolation. This process assigns a grayscale value to each vertex on the mesh, reflecting the density information of the original CT image at that position, so that the final model can intuitively show the internal structure of the tumor and its relationship with the surrounding tissues.
[0062] Finally, the surgical simulation system can save the optimized mesh model and texture information in a standard 3D model format, such as OBJ or PLY. This file contains detailed geometric information and texture values for each vertex. The model can be rotated and scaled through the interface provided by the system to observe the shape, size, position of the tumor, and its relationship with surrounding tissues in all directions. Texture information provides additional diagnostic value to help identify heterogeneity within the tumor or involvement of surrounding tissues.
[0063] It is understandable that other image processing and computer graphics methods may also be used to achieve three-dimensional reconstruction of the lesion site, which is not limited here.
[0064] S102, determining a plurality of optional surgical navigation schemes including different incision positions according to the three-dimensional model of the lesion site, the patient's anatomical structure data, and the positions of important organs; It is understandable that when the surgical simulation system generates the three-dimensional model of the lesion site, it will also generate a three-dimensional anatomical model of the patient as a whole based on the patient's medical imaging data. The generation process is similar to that of generating the three-dimensional model of the lesion site in step S101, and will not be repeated here.
[0065] Patient anatomical data represents detailed information about the anatomical features of individual patients, such as organs, blood vessels, nerves, etc. Vital organ locations refer to the specific locations of key organs related to surgery in the patient's body. Patient anatomical data and vital organ locations are determined by the surgical simulation system from a three-dimensional anatomical model generated based on the patient's medical imaging data.
[0066] The surgical simulation system can load a three-dimensional model of the lesion and spatially register it with the patient's overall three-dimensional anatomical model data. Based on this information, multiple possible incision locations can be generated, each of which corresponds to a complete surgical navigation plan. These plans consider the path from the incision to the lesion and evaluate the feasibility, safety, and effectiveness of each path. The surgical simulation system can score each plan, taking into account factors such as surgical difficulty, impact on surrounding tissues, and surgical time. Ultimately, the surgical simulation system can present multiple optional surgical navigation plans with higher scores for further evaluation and selection.
[0067] Surgical navigation plans can be generated and evaluated in a variety of ways: In some embodiments, the surgical simulation system may adopt a rule-based approach: first, based on preset safety area rules, multiple possible incision locations are determined on the surface of a three-dimensional anatomical model representing the patient's entire body; then, based on a path planning algorithm, one or more paths are designed for each incision location that avoid important organ locations and reach the lesion site in the three-dimensional model of the lesion site; then, based on the patient's anatomical structure data, the length, curvature and distance of each path from the important organ location are calculated, and a path score is obtained according to preset scoring criteria; finally, multiple paths that exceed a preset scoring threshold are determined as multiple optional surgical navigation schemes.
[0068] Specifically, the surgical simulation system first loads the preset safety area rules, such as avoiding the lower edge of the ribs and the area around the umbilicus in abdominal surgery, or giving priority to specific intercostal spaces in thoracic surgery. The preset safety area rules are derived from the experience accumulated in clinical medical practice, anatomical knowledge, and relevant surgical guidelines. These rules comprehensively consider the structural characteristics of various parts of the human body, the distribution of important organs, and the feasibility of surgical operations. The surgical simulation system applies these rules on the surface of the 3D anatomical model, marks the areas that meet the safety standards, and evenly distributes the candidate incision points in these areas, while ensuring that there is enough operating space around each point.
[0069] After determining the potential incision locations, we start designing the path for each incision to reach the lesion. This step can use the improved A* algorithm to discretize the patient's 3D anatomical model into a voxel grid and define a cost function that takes into account distance, tissue type, and safety factors. By fine-tuning the weights of the cost function, multiple different paths can be generated for each incision, providing more options.
[0070] Among them, the A* algorithm is a widely used path search and graph traversal algorithm. This algorithm combines the rigor of the Dijkstra algorithm and the heuristic method of the best-first search, and can efficiently find the optimal path from the starting point to the target point. The A* algorithm uses an evaluation function f(n) to determine the order of search. For each node n, f(n) = g(n) + h(n), where: g(n) is the actual cost from the starting point to the current node n; h(n) is the estimated cost from node n to the target (heuristic function). h(n) is the key part of the algorithm, which estimates the cost from the current node to the target. A good heuristic function can significantly improve the efficiency of the algorithm. In this application, the A* algorithm can be improved to adapt to three-dimensional space and specific medical constraints: the three-dimensional anatomical model of the patient is represented as a three-dimensional voxel grid, and each voxel represents a possible path point. g(n) can contain distance cost, tissue penetration difficulty, etc. h(n) can use straight-line distance in three-dimensional space or more complex estimates considering anatomical structure.
[0071] Each generated path can then be subjected to detailed characterization, which measures the path length, calculates the average curvature, and evaluates the minimum and average distances of the path to important anatomical structures.
[0072] After obtaining the detailed characteristics of all paths, the scheme scoring and sorting stage begins. The predefined scoring criteria are applied to calculate the total score for each scheme, taking into account factors such as path length, curvature, and safety distance. Among them, the predefined scoring criteria can comprehensively consider key factors such as path length (30%), curvature (25%), safety distance from important structures (30%), difficulty of tissue penetration (10%), and surgical field of view (5%), each of which has its specific calculation method and weight. For example, the path length score is calculated by comparing the actual length with the longest acceptable length, while the safety distance score is based on the minimum distance between the path and the key anatomical structure. Then, all schemes are sorted in descending order according to the total score, and the ones with the highest scores are selected as the final optional surgical navigation schemes.
[0073] In some embodiments, the surgical simulation system may also adopt a machine learning-based method: first, a deep neural network model is trained using a large amount of historical surgical data; then, the three-dimensional model of the current patient's lesion, anatomical structure data, and the location of important organs are input into the trained model; then, the model will automatically generate multiple surgical navigation plans ranked from high to low in terms of scores, including incision locations and path planning.
[0074] Specifically, large-scale data collection and preprocessing are first carried out to collect detailed information on a large number of historical surgical cases, including the patient's three-dimensional anatomical structure, lesion characteristics, actual surgical plans and their results. After being standardized and quantified, these rich data are used to train a deep neural network, which includes: 3D convolutional neural network (CNN): processing the patient's anatomical structure data; point cloud processing network (such as PointNet): processing the characteristics of the lesion site; fully connected layer: integrating various features and generating outputs; attention mechanism: focusing on important anatomical structures and potential risk areas. The output of the network can include: heat map of incision location, key point sequence of the path, surgical risk score and expected operation time, etc.
[0075] During the model training phase, 80% of the data can be used as a training set, 10% as a validation set, and 10% as a test set. Define a multi-task loss function: L = λ1L_cut + λ2L_path + λ3L_risk + λ4L_time, where each sub-loss uses an appropriate metric (such as cross entropy for cut positions and average distance error for paths). Adam optimizer is used for training, with an initial learning rate of 0.001 and a cosine annealing strategy. Dropout (ratio 0.5) and L2 regularization (coefficient 0.0001) are applied to prevent overfitting. The model is trained using the PyTorch framework, evaluated on the validation set every 100 batches, and the model with the lowest validation loss is selected as the final model.
[0076] After the model training is completed, the input data is prepared using the same preprocessing process. During model inference, ONNXRuntime can be used for acceleration to generate the 10 most likely incision locations and corresponding paths as multiple surgical navigation solutions presented in the end. For each generated path, the Savitzky-Golay filter can also be applied for smoothing, and the Fast Marching Method can be used to ensure that the path does not pass through restricted areas (such as important blood vessels).
[0077] It is understandable that other methods may be used to generate and evaluate surgical navigation plans, such as combining expert systems or hybrid methods, which are not limited here.
[0078] S103, performing virtual reality simulated surgery according to the first surgical navigation scheme selected by the user, and during the simulated surgery, displaying the surgical field of view according to the user's actual viewing angle based on the user's operation instructions; Among them, operation instructions refer to control commands issued by users through specific input devices (such as handles, data gloves, etc.), which are used to manipulate surgical instruments in a virtual environment.
[0079] In some embodiments, the virtual reality simulation surgery can be performed through a head-mounted VR headset and a controller; in some embodiments, the virtual reality simulation surgery can also be performed through a 3D display and an operating handle, which is not limited here.
[0080] Taking the use of a head-mounted VR headset for virtual reality simulation surgery as an example, first load the relevant data of the first surgical navigation scheme selected by the user, including the patient's three-dimensional anatomical model, the predetermined incision location, and the surgical path. Then, the system initializes the virtual reality environment and converts the loaded data into a format that can be rendered in VR. The user puts on the VR headset and picks up the controller to enter the virtual operating room. The system sets the user's position and orientation in the virtual environment according to the initial perspective of the navigation scheme. Subsequently, the user can issue operation instructions through the controller, such as moving the perspective, selecting and using virtual surgical instruments, etc. The system captures these operation instructions in real time and updates the scene and object status in the virtual environment. At the same time, the system continuously tracks the user's head movements and dynamically adjusts the rendering perspective to ensure that the displayed content is always consistent with the user's actual perspective. Throughout the process, the system can also simulate situations that may be encountered in real surgery, such as bleeding, tissue deformation, etc., to enhance the authenticity of the simulation and the training effect.
[0081] S104, in the case of a simulated surgical operation failure, displaying the three-dimensional field of view of the entire process of the first operation that caused the surgical failure, and resetting the simulated surgical progress to before the first operation was performed; Among them, simulated surgical operation refers to a series of operational behaviors that simulate the real surgical process in a virtual reality environment. Surgical failure means that the results in the virtual surgical process are not in line with expectations or may endanger the safety of the patient. The first operation refers to the key operation step that leads to the failure of the operation. The full process three-dimensional field of view is used to represent the complete three-dimensional picture of the first operation from multiple angles and time points. Resetting the progress of the simulated surgery means rolling back the state of the virtual surgical system to the time point before the first operation.
[0082] Specifically, the surgical simulation system first determines whether the operation has failed through preset evaluation criteria (such as organ damage, bleeding volume, operation time, etc.). Once the failure is confirmed, the entire surgical process will be traced back to analyze the key operations that led to the failure. Then, the surgical simulation system calls the complete three-dimensional record of the operation, including the operator's perspective, hand movements, instrument trajectory and other information. This information is integrated into a full range of three-dimensional vision and displayed on the user interface. At the same time, the system can mark the key points of error and provide a comparative view of the correct operation. Finally, the state of the virtual surgery is automatically rolled back to before the first operation that caused the failure, and all relevant data at that time point are reloaded, including the patient's physiological state, surgical progress, instrument position, etc., to provide users with the opportunity to try again.
[0083] In some embodiments, an expert system knowledge base containing a large amount of surgical knowledge and experience can be pre-built, which defines the standard procedures for various operations, the allowable deviation range, and potential risk factors. When the operation fails, the system starts from the failure result and uses the decision tree algorithm to trace the possible causes step by step. At each decision node, the key indicators of the current state (such as the degree of organ damage, bleeding volume, vital signs, etc.) are evaluated and compared with the thresholds in the knowledge base. At the same time, the time sequence and cumulative effects of the operations are considered, and the degree of influence of each operation on these indicators is calculated. When the earliest operation that causes the key indicator to exceed the safety threshold is found, it is determined as the first operation that caused the failure.
[0084] In some embodiments, a deep learning model can be used to monitor every step of the user's operation in real time and compare it with standard operating specifications. When a behavior that seriously deviates from the standard operation is detected, a failure alarm is immediately triggered and the key operation sequence that led to the failure is recorded. Computer vision technology is used to capture the entire process of the erroneous operation from multiple virtual camera angles to generate high-precision three-dimensional reconstruction data. Natural language processing technology is applied to automatically generate text descriptions and voice explanations of the erroneous operation, and provide a full range of error analysis in conjunction with visual replay.
[0085] In some embodiments, an accurate physical model of the surgical environment can be constructed first, including tissue deformation, fluid mechanics, and instrument interaction; during the simulation process, changes in all physical parameters are continuously recorded to form a complete state history log; when a surgical failure is detected, the physical engine is used for reverse deduction to accurately locate the initial erroneous operation that caused the failure; based on the state history log, an accurate rollback of the surgical environment is achieved, restoring all relevant parameters to the state before the error occurred.
[0086] It is understandable that other methods may be used to realize the display of erroneous operations and the resetting of progress, such as combining virtual reality tactile feedback to enhance the perception of erroneous operations, or integrating expert systems to provide personalized improvement suggestions, etc., which are not limited here.
[0087] S105. When the number of failures of the simulated surgical operation for the first operation exceeds a preset upper limit of failures, display a recommended second surgical navigation plan, where the second surgical navigation plan is a surgical navigation plan that does not include the first operation among the multiple optional surgical navigation plans.
[0088] The number of simulated surgical operation failures refers to the cumulative number of times a specific surgical step is repeated in a virtual environment and results in failure. The preset upper limit of failures refers to the maximum number of failures allowed by the system, which is used to determine whether further intervention measures are needed.
[0089] Specifically, the surgical simulation system can continuously monitor the number of failures of the operator in a specific step (i.e., the first operation). After each failure, the system will record and update the failure count. When the number of failures reaches the preset upper limit, the system can trigger the scheme switching mechanism. At this time, the system can filter out the scheme that does not contain the first operation that caused repeated failures from the multiple optional surgical navigation schemes prepared in advance, and display it as the second surgical navigation scheme for the user to choose. Optionally, the system can also provide a comparison view to show the differences between the new and old schemes to help the operator understand the reasons and content of the adjustments. After the user selects a new navigation scheme, the simulation training can be restarted based on the new navigation scheme.
[0090] In some embodiments, a rule-based scheme screening method can be adopted: a series of rules are pre-set to define the applicable conditions and priorities of different surgical navigation schemes; when the number of failures reaches the upper limit, the system automatically triggers the rule engine; the rule engine first excludes all schemes containing failed operations; then the remaining schemes are scored according to preset rules, such as difficulty level, operator experience, etc.; the scheme with the highest score is selected as the recommended second surgical navigation scheme and displayed to the user.
[0091] In some embodiments, a case-based reasoning approach to recommending surgical plans can be used: a database containing a large number of historical surgical cases is established, each case containing information such as surgical plans, operator conditions, and success rates; when a new plan needs to be recommended, the system analyzes the characteristics of the current operator and the cause of failure; a similarity algorithm is used to search the case library for successful cases that are most similar to the current situation; surgical plans that do not contain failed operations are extracted from these similar successful cases; these extracted surgical plans that do not contain failed operations are matched with multiple optional surgical navigation plans, and the surgical navigation plan with a high degree of match is displayed to the user as the second surgical navigation plan.
[0092] In the embodiment of the present application, the precise three-dimensional model of the lesion site generated based on the patient's medical imaging data, combined with the patient's individualized anatomical structure data, enables the doctor to intuitively and comprehensively understand the spatial position of the lesion and the relationship between the surrounding important tissues, overcoming the difficulty of inaccurate grasp of the spatial relationship of complex cases in traditional methods. The simulated surgery performed in a virtual reality environment provides doctors with an immersive training experience, and the full-process three-dimensional field of view playback and progress reset function when the operation fails provides doctors with timely and effective error correction opportunities, significantly improving the pertinence and interactivity of the training. The mechanism of intelligently recommending alternative surgical navigation plans in the event of repeated failures enhances the flexibility and adaptability of the surgical plan and effectively responds to complex and changeable surgical situations. The organic combination of these technical features enables doctors to fully understand the three-dimensional structure of the lesion site before surgery and obtain targeted operational training, improving the effectiveness of surgical simulation.
[0093] In the above embodiments, the surgical simulation system can achieve the technical effect of improving the accuracy of surgical plan formulation and surgical safety by integrating 3D reconstruction of patient medical imaging data, multi-plan surgical navigation and virtual reality simulation technology. However, in actual applications, the selection and coordination of assistant surgeons are also key factors for the success of surgery, and the surgical simulation system can also provide effective assistance.
[0094] See also Figure 2 , is another flow chart of the surgical simulation method based on three-dimensional image reconstruction and surgical navigation in an embodiment of the present application.
[0095] S201, generating a three-dimensional model of the lesion site through image segmentation and three-dimensional reconstruction based on the patient's medical imaging data; S202, determining a plurality of optional surgical navigation schemes including different incision positions according to the three-dimensional model of the lesion site, the patient's anatomical structure data, and the positions of important organs; S203, performing virtual reality simulated surgery according to the first surgical navigation scheme selected by the user, and during the simulated surgery, displaying the surgical field of view according to the user's actual viewing angle based on the user's operating instructions; S204, in the case of a failure in the simulated surgical operation, displaying the three-dimensional field of view of the entire process of the first operation that caused the failure of the operation, and resetting the progress of the simulated surgery to before the first operation was performed; S205. When the number of failures of the simulated surgical operation for the first operation exceeds a preset upper limit of failures, display a recommended second surgical navigation plan, where the second surgical navigation plan is a surgical navigation plan that does not include the first operation among the multiple optional surgical navigation plans.
[0096] Steps S201 to S205 are similar to steps S101 to S105 and will not be described again here.
[0097] S206, receiving a combination of multiple assistant surgeons in the hospital input by a user; Among them, assistant surgeons refer to medical staff who assist the surgeon in completing the operation, usually including assistant doctors, instrument nurses, etc. The combination of assistant surgeons refers to a team composed of multiple assistant surgeons.
[0098] Specifically, before the virtual reality simulation surgery begins, the assistant surgeon selection stage can be entered. The surgical simulation system can display an assistant surgeon selection interface, listing all available assistant surgeons in the hospital, including their names, titles, expertise and other information. Users can select multiple assistant surgeons to form a combination through interactive elements such as drop-down menus or multiple-selection boxes on the interface. Users can also be allowed to create multiple different assistant surgeon combinations for subsequent evaluation and comparison. Each time a user completes the selection of a combination and confirms it, the combination information can be saved in a temporary database. Users can repeat this process as needed to create multiple different assistant surgeon combinations.
[0099] In some embodiments, the input and management of the assistant personnel combination can be achieved in multiple ways: Optionally, a graphical drag-and-drop interface can be provided: the avatars and brief information of all available auxiliary knife personnel are displayed on the left side of the interface. Multiple blank "combination areas" are set on the right side, each area represents an auxiliary knife personnel combination. Users can create a combination by dragging the avatar on the left to the combination area on the right. Each combination area has a "Save" button, which saves the combination information to the system after clicking it. Users can modify the created combination at any time, such as adding or deleting members.
[0100] Optionally, a role-based combination generator can also be provided: first define the common auxiliary roles in surgery, such as "first assistant", "second assistant", "instrument nurse", etc. The user selects the required role combination, such as "2 assistants + 1 instrument nurse". The system automatically filters out qualified candidates from the available personnel based on the selected role combination. The user selects specific personnel for each role from the candidates. After completing the selection, the system automatically generates an auxiliary personnel combination and provides a save option.
[0101] It is understandable that other methods can also be used to input and manage the combination of auxiliary surgeons, such as oral input based on voice recognition, or intelligent recommendation systems combined with artificial intelligence, etc., which are not limited here.
[0102] S207, based on the historical surgery database, determining the success rate of each auxiliary surgeon combination participating in each optional surgical navigation scheme; Among them, the historical surgery database refers to a storage system containing past surgery records and related information, which is used to analyze and predict the success rate of future surgeries.
[0103] Specifically, after receiving multiple assistant surgeon combinations input by the user, the historical surgery database can be accessed to extract all records related to the current surgery type. Then, for each assistant surgeon combination input by the user, the system will search the historical data for relevant surgery records in which the combination (or similar combinations) participated. At the same time, the system will match these historical records with the current optional surgical navigation schemes. By analyzing the matched historical data, the system calculates the success rate of each assistant surgeon combination in each optional surgical navigation scheme. This process takes into account multiple factors, such as the complexity of the surgery, the experience level of the assistant surgeon, the degree of tacit understanding of previous cooperation, etc. Finally, the system generates a matrix showing the expected success rate of each assistant surgeon combination in each surgical navigation scheme.
[0104] In some embodiments, the analysis of surgical success rate based on historical data can be achieved in a variety of ways: Optionally, the system can use a rule-based analysis method: First, define a series of attributes for each assistant surgeon, such as years of experience, professional field, skill score, etc. Then, set the difficulty coefficient and key skill requirements for each surgical navigation plan. The system screens out cases similar to the current type of surgery from the historical database. For each combination of assistant surgeon and surgical navigation plan, the system calculates a matching score, considering the fit between the assistant surgeon's attributes and the plan requirements. Finally, based on the matching score and the proportion of historical successful cases, the expected success rate is calculated.
[0105] Optionally, the system can also use a machine learning model to predict success rates: Collect and preprocess historical surgical data, including assistant surgeon information, surgical plan details, and surgical results. Use this data to train a machine learning model, such as a random forest or gradient boosting tree. For each new assistant surgeon combination and surgical navigation plan, input its features into the trained model. The model outputs the predicted success rate of the combination under a specific plan. Repeat this process to generate success rate predictions for all combinations and plans.
[0106] It is understandable that other methods can also be used to analyze and predict the success rate of surgery, such as using a Bayesian network model to consider the complex dependencies between factors, or combining an expert system with the experience and judgment of medical experts, etc., which are not limited here.
[0107] S208, when displaying the recommended second surgical navigation plan, sorting the surgical navigation plans according to the patient's planned surgery time, the time schedule of each auxiliary surgeon combination in the current hospital, and the success rate; The planned surgery time of the patient refers to the specific date and time period for the scheduled surgery. The time schedule of each auxiliary surgery personnel combination in the current hospital refers to the work arrangement of each potential auxiliary surgery team in the future.
[0108] When showing the recommended second surgical navigation plan to the user, the patient's planned surgery time can be retrieved, and then the recent work schedule of each auxiliary knife personnel combination in the current hospital can be queried to determine which combinations are available at the planned surgery time. Next, combined with the previously calculated success rates of each combination under different plans, the feasibility and potential effect of each optional surgical navigation plan are comprehensively evaluated. During the evaluation process, a weighted algorithm can be used to comprehensively consider factors such as time matching, personnel availability, and expected success rate. Finally, all feasible surgical navigation plans are sorted according to these comprehensive scores and displayed to the user according to the sorting results. At the same time, the reasons for the sorting of each optional surgical navigation plan can be provided.
[0109] In some embodiments, intelligent sorting of surgical navigation solutions can be achieved in a variety of ways: Optionally, a multi-factor weighted scoring method can be used: assign weights to each factor (time matching, personnel availability, expected success rate), such as 30% for time matching, 30% for personnel availability, and 40% for expected success rate. For time matching, calculate the degree of overlap between the time required for the plan and the patient's planned surgery time. Full marks are awarded for complete matching, and scores are scored proportionally for partial overlap. For personnel availability, check the idleness of each auxiliary knife personnel combination during the planned time period. Full marks are awarded for full availability, and scores are scored proportionally for partial availability. The expected success rate is directly used as the score for this item. Add the three scores according to the weights to obtain the total score of each plan, and sort them accordingly.
[0110] Optionally, you can use a decision tree algorithm for sorting: Build a decision tree with time matching, personnel availability, and expected success rate as decision nodes. At the first level of the tree, classify the solutions into three categories: "complete match", "partial match", and "mismatch" based on time matching. At the second level, classify each category into "all available", "partially available", and "all unavailable" based on personnel availability. At the third level, use the expected success rate for final sorting. Traverse the decision tree and sort all solutions from best to worst.
[0111] It is understandable that other methods may be used to achieve the ranking of surgical navigation solutions, such as using fuzzy logic to consider the complex relationship between various factors, or applying genetic algorithms to dynamically optimize the ranking results, etc., which are not limited here.
[0112] In some embodiments, each time the optional surgical navigation schemes are displayed, including the first time the plurality of optional surgical navigation schemes are displayed, the displayed navigation schemes may be sorted according to steps S206 to S208, which is not limited here.
[0113] S209, when the user selects a surgical navigation scheme to perform virtual reality simulated surgery, adjusting the list of assistant surgeons during the simulated surgery according to the user's modification operation; After the user selects one of the surgical navigation schemes recommended by the system and starts the virtual reality simulated surgery, at the beginning of the simulated surgery, the system will display the default list of assistant surgeons according to the previous settings. However, the user may want to adjust the assistant surgeons according to the actual situation or personal preferences. Specifically, the system can provide an interactive interface that allows the user to view the current list of assistant surgeons and modify them. The user can add new assistant surgeons, remove existing personnel, or replace personnel of specific roles through this interface. Every time the user makes a modification, the system will respond immediately and update the assistant surgeon settings in the virtual environment. This may include updating the appearance of the virtual characters, adjusting their positions in the operating room, and even changing their behavior patterns in the simulation. The system can also calculate and display in real time the possible impact of these changes on the success rate of the operation to help users make decisions. The user can make adjustments multiple times until he is satisfied with the configuration of the assistant surgeons and then continue the simulated surgery.
[0114] In some embodiments, dynamic adjustment of assistant surgeons in simulated surgery can be achieved in a variety of ways: Optionally, the system can provide a graphical drag-and-drop interface: a translucent personnel management panel is displayed on one side of the virtual operating room scene. The panel lists the current assistant surgeons, and there is a "Remove" button next to each person. There is an "Add Personnel" button at the bottom of the panel, which pops up a list of available personnel. Users can adjust the position of personnel in the operating room by dragging. After each adjustment, the system updates the personnel configuration and position in the virtual environment in real time.
[0115] Optionally, the system can also use voice commands and gesture control: users can modify the personnel list through voice commands such as "add instrument nurse Zhang San". After the system recognizes the command, it adds the corresponding role model in the virtual environment. Users can point to a virtual character through gestures and then make a "cross out" action to remove the person. The system will confirm each operation through voice feedback, such as "Instrument nurse Zhang San has been added". Users can adjust the position or responsibilities of the virtual character through specific gesture combinations.
[0116] It is understandable that other methods can also be used to achieve dynamic adjustment of assistant surgeons, such as using augmented reality technology to add and remove virtual personnel in a real operating room environment, etc., which is not limited here.
[0117] S210. During the simulated surgery, based on the historical operation records of the assistant surgeon in the historical surgery database, the assistant surgeon's corresponding operations are completed by self-matching.
[0118] Among them, the historical operation records of the assistant surgeon refer to the detailed records of the specific behaviors and performances of each assistant surgeon in the past operations. Self-matching refers to the system automatically selecting and executing operations according to the personal operation habits of the assistant surgeon.
[0119] During the virtual reality simulation of surgery, the system first identifies the current surgical stage and specific steps. For each virtual assistant, the historical operation records of that particular person are queried in the historical surgery database, and these historical records are analyzed to extract the typical behavior patterns and personal habits of the assistant in similar surgical stages. This may include their preferred standing position, instrument delivery method, reaction speed, etc. Next, the system generates a corresponding operation sequence for each assistant in the virtual environment based on the extracted personal operation habits. These operations may not always perfectly match the main surgeon, but truly reflect the personal characteristics of each assistant. Finally, the system performs these self-matching auxiliary operations in the virtual reality environment, so that the behavior of each virtual assistant appears to be in line with his or her personal style. This method allows the user (the main surgeon) to experience the challenges of working with assistants with different characteristics during training, thereby improving adaptability and coordination.
[0120] In some embodiments, the self-matching of the assistant operator's operation based on personal habits can be achieved in a variety of ways: Optionally, the system can use a statistically based personal habit model: perform data mining on the historical operation records of each assistant surgeon to extract frequent operation patterns and time distributions. Establish an operation probability distribution model for the assistant surgeon for each type and stage of surgery. In simulated surgery, the system identifies the current surgical stage and randomly samples from the corresponding probability distribution to determine the next operation. The system considers the time interval between operations and simulates the reaction speed and rhythm of the assistant surgeon. Execute the sampled operations and record the operation results for subsequent model updates and optimizations.
[0121] Optionally, the system can also use a personalized reinforcement learning model: train a unique reinforcement learning agent for each assistant, with the initial strategy based on their historical operation data. Define the state space (including surgical stage, instrument position, etc.) and action space (various possible auxiliary operations). During the simulation, the agent selects actions based on the current state, but adds a certain amount of randomness to simulate human uncertainty. The system executes the selected action and gives rewards based on whether it conforms to the assistant's historical habits. The agent continues to learn and adapt, gradually forming a behavior pattern that is more in line with individual characteristics.
[0122] It is understandable that other methods can also be used to achieve self-matching of assistant knife personnel's operations based on personal habits, such as using a generative adversarial network (GAN) to generate an operation sequence that conforms to the style of a specific assistant knife personnel, or combining a fuzzy logic system to simulate the uncertainty of human decision-making, etc., which are not limited here.
[0123] In some embodiments, steps S209-S210 may not be performed; in some embodiments, steps S209-S210 may be performed any time the user selects a surgical navigation plan to perform virtual reality simulated surgery, which is not limited here.
[0124] In the embodiment of the present application, due to the use of assistant surgeon combination evaluation and personalized operation simulation technology based on historical data, it is possible to help users select the best assistant team and truly simulate the personal operating habits of each assistant surgeon in a virtual environment, thereby improving the authenticity and effectiveness of surgical simulation training, making it possible to comprehensively improve the quality of surgical rehearsal, provide more comprehensive, accurate, and personalized preoperative preparation, and ultimately achieve the goal of reducing surgical risks.
[0125] The following introduces an exemplary surgical simulation system 100 provided in an embodiment of the present application. Figure 3 Schematic diagram of an exemplary hardware structure of the surgical simulation system 100 provided in an embodiment of the present application.
[0126] In some embodiments, the surgical simulation system 100 includes a computer device. The computer device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The network interface of the computer device is used to communicate with other external terminals or servers through a network connection. In some embodiments, the network interface can be a wired network interface, and in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, the method in the embodiment of the present application is implemented.
[0127] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0128] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.
[0129] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.
[0130] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk), etc.
[0131] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
Claims
1. A surgical simulation method based on three-dimensional image reconstruction and surgical navigation, characterized in that: include: Based on the patient's medical imaging data, a three-dimensional model of the lesion is generated through image segmentation and three-dimensional reconstruction; Determine multiple optional surgical navigation schemes including different incision positions according to the three-dimensional model of the lesion site, the patient's anatomical structure data, and the position of important organs; Perform virtual reality simulated surgery according to the first surgical navigation scheme selected by the user. During the simulated surgery, the surgical field of view is displayed according to the user's actual perspective based on the user's operation instructions. In the case of a simulated surgical operation failure, the three-dimensional field of view of the entire process of the first operation that caused the surgical failure is displayed, and the simulated surgical progress is reset to before the first operation is performed; When the number of failures of the simulated surgical operation for the first operation exceeds a preset upper limit of failures, a recommended second surgical navigation plan is displayed, where the second surgical navigation plan is a surgical navigation plan that does not include the first operation among the multiple optional surgical navigation plans.
2. The method according to claim 1, characterized in that Determining a plurality of optional surgical navigation schemes including different incision positions according to the three-dimensional model of the lesion site, the patient's anatomical structure data and the position of important organs specifically includes: According to preset safety area rules, multiple possible incision positions are determined on the surface of the three-dimensional anatomical model representing the whole body of the patient; Based on the path planning algorithm, one or more paths are designed for each incision position, avoiding the positions of important organs and reaching the lesion site in the three-dimensional model of the lesion site; Based on the patient's anatomical structure data, the length, curvature and distance to the location of important organs of each path are calculated, and a path score is obtained according to a preset scoring standard; A plurality of paths exceeding a preset score threshold are determined as the plurality of optional surgical navigation schemes.
3. The method according to claim 1, characterized in that The display of the surgical field of view according to the actual viewing angle of the user specifically includes: Collecting the user's head position, rotation angle or eye movement data in real time to obtain viewing angle information; wherein the viewing angle information refers to the user's viewing position and viewing direction in the virtual environment, the viewing position is determined by the head position, and the viewing direction is determined by the rotation angle or eye movement data; According to the viewing angle information, update the virtual camera parameters, wherein the virtual camera parameters include the position and orientation of the camera in the virtual environment; wherein the virtual camera refers to an observation point that simulates a real camera in the three-dimensional virtual environment; The surgical scene is re-rendered using the updated virtual camera parameters and pushed for display.
4. The method according to claim 1, characterized in that: The method of displaying the full three-dimensional field of view of the first operation that caused the failure of the operation in the case of a simulated failure of the operation specifically includes: In the case where the surgical operation is determined to have failed based on the preset evaluation criteria, the entire surgical process is reviewed to determine the first operation that caused the surgical failure; Recalling a complete three-dimensional record of the first operation, including the operator's perspective, hand movements, and instrument trajectories; Based on the complete three-dimensional record of the first operation, a three-dimensional field of view of the entire process of the first operation is generated and displayed on a user interface.
5. The method according to claim 1, characterized in that The method further comprises: After the user selects a next surgical navigation plan from the recommended displayed second surgical navigation plans, displaying a comparison view, wherein the comparison view includes the difference between the next surgical navigation plan and the first surgical navigation plan; The virtual reality simulated surgery is reset based on the next surgical navigation plan.
6. The method according to any one of claims 1 to 5, characterized in that The scheme also includes: When the user selects a surgical navigation scheme to perform a virtual reality simulated surgery, the list of assistant surgeons during the simulated surgery is adjusted according to the user's modification operation; During the simulated surgery, the corresponding operations of the assistant surgeon are completed through self-matching based on the historical operation records of the assistant surgeon in the historical surgery database.
7. The method according to claim 6, characterized in that The method further comprises: Receive multiple combinations of assistant surgeons in the hospital input by a user; Based on the historical surgery database, determining the success rate of each auxiliary surgeon combination participating in each optional surgical navigation scheme; When displaying the recommended second surgical navigation plan, the surgical navigation plans are sorted according to the patient's planned surgery time, the time schedule of each auxiliary surgeon combination in the current hospital, and the success rate.
8. A surgical simulation system, characterized in that: The surgical simulation system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the surgical simulation system to execute the method described in any one of claims 1-7.
9. A computer program product comprising instructions, characterized in that When the computer program product runs on a surgery simulation system, the surgery simulation system is caused to perform the method according to any one of claims 1 to 7.
10. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a surgical simulation system, the surgical simulation system is caused to perform the method according to any one of claims 1 to 7.
Citation Information
Patent Citations
Virtual reality and medical image-based surgery analogy method, device and equipment
CN106974730A
Method of setting up virtual reality medical team
CN108538171A
Beam assisted positioning method for craniotomy and positioning system of beam assisted positioning method for craniotomy
CN111728695A
Virtual reality-based minimally invasive surgery simulation method and system
CN113197665A
Cardiothoracic surgery auxiliary control system based on virtual reality
CN113274129A
Cited By
Thyroid surgery patient clinical analysis system and method based on 3D image
CN119606530A
Clinical analysis system and method for thyroid surgery patients based on 3D images
CN119606530B
Surgical minimally invasive surgery data processing method and system based on AR technology
CN119811686A
Individualized virtual laparoscopic surgery robot platform and device
CN120048473A
Intraoral dynamic occlusal force model generation method and device, electronic equipment and storage medium
CN121306578A
